2022-03-17T19:03:26.4150585Z Requested labels: linux.rocm.gpu 2022-03-17T19:03:26.4150723Z Job defined at: pytorch/pytorch/.github/workflows/generated-linux-bionic-rocm4.5-py3.7-distributed.yml@refs/heads/master 2022-03-17T19:03:26.4150750Z Waiting for a runner to pick up this job... 2022-03-17T19:03:27.9360723Z Job is about to start running on the runner: worker-rocm-amd-34 (repository) 2022-03-17T19:03:32.2187478Z Current runner version: '2.288.1' 2022-03-17T19:03:32.2192856Z Runner name: 'worker-rocm-amd-34' 2022-03-17T19:03:32.2193344Z Runner group name: 'Default' 2022-03-17T19:03:32.2194050Z Machine name: 'jenkins-worker-rocm-amd-34' 2022-03-17T19:03:32.2196168Z ##[group]GITHUB_TOKEN Permissions 2022-03-17T19:03:32.2196758Z Actions: write 2022-03-17T19:03:32.2197036Z Checks: write 2022-03-17T19:03:32.2197418Z Contents: write 2022-03-17T19:03:32.2197702Z Deployments: write 2022-03-17T19:03:32.2198041Z Discussions: write 2022-03-17T19:03:32.2198313Z Issues: write 2022-03-17T19:03:32.2198602Z Metadata: read 2022-03-17T19:03:32.2198909Z Packages: write 2022-03-17T19:03:32.2199181Z Pages: write 2022-03-17T19:03:32.2199501Z PullRequests: write 2022-03-17T19:03:32.2199818Z RepositoryProjects: write 2022-03-17T19:03:32.2200205Z SecurityEvents: write 2022-03-17T19:03:32.2200496Z Statuses: write 2022-03-17T19:03:32.2200790Z ##[endgroup] 2022-03-17T19:03:32.2204412Z Secret source: Actions 2022-03-17T19:03:32.2205095Z Prepare workflow directory 2022-03-17T19:03:32.3533921Z Prepare all required actions 2022-03-17T19:03:32.3795387Z Getting action download info 2022-03-17T19:03:32.7129422Z Download action repository 'zhouzhuojie/checkout@05b13c9a0d21f08f6d5e64a1d5042246d13619d9' (SHA:05b13c9a0d21f08f6d5e64a1d5042246d13619d9) 2022-03-17T19:03:33.9125240Z Download action repository 'seemethere/download-artifact-s3@0504774707cbc8603d7dca922e8026eb8bf3b47b' (SHA:0504774707cbc8603d7dca922e8026eb8bf3b47b) 2022-03-17T19:03:35.1583178Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2) 2022-03-17T19:03:36.3791992Z ##[group]Run rm -rf "${GITHUB_WORKSPACE}" 2022-03-17T19:03:36.3792444Z rm -rf "${GITHUB_WORKSPACE}" 2022-03-17T19:03:36.3792873Z mkdir "${GITHUB_WORKSPACE}" 2022-03-17T19:03:36.3828993Z shell: /bin/bash -e {0} 2022-03-17T19:03:36.3829230Z env: 2022-03-17T19:03:36.3829576Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:36.3830032Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:36.3830503Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:36.3830934Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:36.3831270Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:36.3831536Z IN_CI: 1 2022-03-17T19:03:36.3831755Z IS_GHA: 1 2022-03-17T19:03:36.3831985Z IN_WHEEL_TEST: 1 2022-03-17T19:03:36.3832290Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:36.3832710Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:36.3833032Z PR_LABELS: [] 2022-03-17T19:03:36.3833583Z GITHUB_TOKEN: *** 2022-03-17T19:03:36.3833873Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:36.3834112Z PR_NUMBER: 2022-03-17T19:03:36.3834458Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:36.3834757Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:36.3835041Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:36.3835516Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:36.3836013Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:36.3836347Z TEST_CONFIG: distributed 2022-03-17T19:03:36.3836600Z SHARD_NUMBER: 1 2022-03-17T19:03:36.3836868Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:36.3837134Z PR_BODY: 2022-03-17T19:03:36.3837389Z ##[endgroup] 2022-03-17T19:03:37.8470495Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-03-17T19:03:37.8470962Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-03-17T19:03:37.8503665Z shell: /bin/bash -e {0} 2022-03-17T19:03:37.8503879Z env: 2022-03-17T19:03:37.8504171Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:37.8504612Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:37.8505164Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:37.8505556Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:37.8505885Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:37.8506118Z IN_CI: 1 2022-03-17T19:03:37.8506321Z IS_GHA: 1 2022-03-17T19:03:37.8506531Z IN_WHEEL_TEST: 1 2022-03-17T19:03:37.8506814Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:37.8507199Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:37.8507494Z PR_LABELS: [] 2022-03-17T19:03:37.8507926Z GITHUB_TOKEN: *** 2022-03-17T19:03:37.8508161Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:37.8508393Z PR_NUMBER: 2022-03-17T19:03:37.8508629Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:37.8508887Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:37.8509136Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:37.8509566Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:37.8510054Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:37.8510360Z TEST_CONFIG: distributed 2022-03-17T19:03:37.8510578Z SHARD_NUMBER: 1 2022-03-17T19:03:37.8510793Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:37.8510999Z PR_BODY: 2022-03-17T19:03:37.8511200Z ##[endgroup] 2022-03-17T19:03:37.8634361Z ##[group]Run cat /etc/os-release || true 2022-03-17T19:03:37.8634939Z cat /etc/os-release || true 2022-03-17T19:03:37.8635480Z cat /etc/apt/sources.list.d/rocm.list || true 2022-03-17T19:03:37.8636051Z cat /opt/rocm/.info/version || true 2022-03-17T19:03:37.8636494Z whoami 2022-03-17T19:03:37.8695918Z shell: /bin/bash -e {0} 2022-03-17T19:03:37.8696326Z env: 2022-03-17T19:03:37.8696847Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:37.8697709Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:37.8698509Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:37.8699236Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:37.8699859Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:37.8700263Z IN_CI: 1 2022-03-17T19:03:37.8700632Z IS_GHA: 1 2022-03-17T19:03:37.8701050Z IN_WHEEL_TEST: 1 2022-03-17T19:03:37.8701579Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:37.8702247Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:37.8702787Z PR_LABELS: [] 2022-03-17T19:03:37.8703589Z GITHUB_TOKEN: *** 2022-03-17T19:03:37.8704038Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:37.8704459Z PR_NUMBER: 2022-03-17T19:03:37.8704914Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:37.8705382Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:37.8705859Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:37.8706670Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:37.8707576Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:37.8708144Z TEST_CONFIG: distributed 2022-03-17T19:03:37.8708556Z SHARD_NUMBER: 1 2022-03-17T19:03:37.8708966Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:37.8709379Z PR_BODY: 2022-03-17T19:03:37.8709847Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:03:37.8710312Z ##[endgroup] 2022-03-17T19:03:37.8888030Z NAME="Ubuntu" 2022-03-17T19:03:37.8888777Z VERSION="18.04.5 LTS (Bionic Beaver)" 2022-03-17T19:03:37.8889408Z ID=ubuntu 2022-03-17T19:03:37.8889939Z ID_LIKE=debian 2022-03-17T19:03:37.8890566Z PRETTY_NAME="Ubuntu 18.04.5 LTS" 2022-03-17T19:03:37.8891190Z VERSION_ID="18.04" 2022-03-17T19:03:37.8891915Z HOME_URL="https://www.ubuntu.com/" 2022-03-17T19:03:37.8892821Z SUPPORT_URL="https://help.ubuntu.com/" 2022-03-17T19:03:37.8893200Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-17T19:03:37.8893854Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-17T19:03:37.8894198Z VERSION_CODENAME=bionic 2022-03-17T19:03:37.8894430Z UBUNTU_CODENAME=bionic 2022-03-17T19:03:37.8894719Z deb [arch=amd64] http://repo.radeon.com/rocm/apt/4.2 xenial main 2022-03-17T19:03:37.8895034Z 4.2.0-21 2022-03-17T19:03:37.8895235Z pytorchci 2022-03-17T19:03:37.8912463Z ##[group]Run rocm-smi 2022-03-17T19:03:37.8912706Z rocm-smi 2022-03-17T19:03:37.8945274Z shell: /bin/bash -e {0} 2022-03-17T19:03:37.8945484Z env: 2022-03-17T19:03:37.8945757Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:37.8946199Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:37.8946628Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:37.8947008Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:37.8947333Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:37.8947542Z IN_CI: 1 2022-03-17T19:03:37.8947734Z IS_GHA: 1 2022-03-17T19:03:37.8947928Z IN_WHEEL_TEST: 1 2022-03-17T19:03:37.8948202Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:37.8948550Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:37.8948830Z PR_LABELS: [] 2022-03-17T19:03:37.8949283Z GITHUB_TOKEN: *** 2022-03-17T19:03:37.8949516Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:37.8949744Z PR_NUMBER: 2022-03-17T19:03:37.8949973Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:37.8950228Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:37.8950476Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:37.8950895Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:37.8951384Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:37.8951685Z TEST_CONFIG: distributed 2022-03-17T19:03:37.8951902Z SHARD_NUMBER: 1 2022-03-17T19:03:37.8952113Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:37.8952315Z PR_BODY: 2022-03-17T19:03:37.8952563Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:03:37.8952797Z ##[endgroup] 2022-03-17T19:03:37.9918114Z 2022-03-17T19:03:37.9919597Z 2022-03-17T19:03:37.9920202Z ======================= ROCm System Management Interface ======================= 2022-03-17T19:03:37.9921279Z ================================= Concise Info ================================= 2022-03-17T19:03:37.9922678Z GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2022-03-17T19:03:37.9923557Z 0 29.0c 16.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-03-17T19:03:37.9924280Z 1 29.0c 16.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-03-17T19:03:37.9924969Z ================================================================================ 2022-03-17T19:03:37.9925682Z ============================= End of ROCm SMI Log ============================== 2022-03-17T19:03:38.0026523Z ##[group]Run rocminfo 2022-03-17T19:03:38.0026974Z rocminfo 2022-03-17T19:03:38.0083479Z shell: /bin/bash -e {0} 2022-03-17T19:03:38.0083886Z env: 2022-03-17T19:03:38.0084411Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:38.0085252Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:38.0086052Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:38.0086778Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:38.0087406Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:38.0087808Z IN_CI: 1 2022-03-17T19:03:38.0088177Z IS_GHA: 1 2022-03-17T19:03:38.0088557Z IN_WHEEL_TEST: 1 2022-03-17T19:03:38.0089089Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:38.0089906Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:38.0090488Z PR_LABELS: [] 2022-03-17T19:03:38.0091342Z GITHUB_TOKEN: *** 2022-03-17T19:03:38.0091798Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:38.0092232Z PR_NUMBER: 2022-03-17T19:03:38.0092678Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:38.0093166Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:38.0093624Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:38.0094436Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:38.0095145Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:38.0095480Z TEST_CONFIG: distributed 2022-03-17T19:03:38.0095709Z SHARD_NUMBER: 1 2022-03-17T19:03:38.0095916Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:38.0096129Z PR_BODY: 2022-03-17T19:03:38.0096375Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:03:38.0096618Z ##[endgroup] 2022-03-17T19:03:38.0999144Z ROCk module is loaded 2022-03-17T19:03:38.0999851Z ===================== 2022-03-17T19:03:38.1000674Z HSA System Attributes 2022-03-17T19:03:38.1001619Z ===================== 2022-03-17T19:03:38.1002380Z Runtime Version: 1.1 2022-03-17T19:03:38.1003056Z System Timestamp Freq.: 1000.000000MHz 2022-03-17T19:03:38.1003934Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-03-17T19:03:38.1004815Z Machine Model: LARGE 2022-03-17T19:03:38.1005574Z System Endianness: LITTLE 2022-03-17T19:03:38.1006023Z 2022-03-17T19:03:38.1006225Z ========== 2022-03-17T19:03:38.1006782Z HSA Agents 2022-03-17T19:03:38.1007333Z ========== 2022-03-17T19:03:38.1007853Z ******* 2022-03-17T19:03:38.1008596Z Agent 1 2022-03-17T19:03:38.1009156Z ******* 2022-03-17T19:03:38.1010114Z Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1011133Z Uuid: CPU-XX 2022-03-17T19:03:38.1012142Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1012939Z Vendor Name: CPU 2022-03-17T19:03:38.1013677Z Feature: None specified 2022-03-17T19:03:38.1014425Z Profile: FULL_PROFILE 2022-03-17T19:03:38.1015175Z Float Round Mode: NEAR 2022-03-17T19:03:38.1015883Z Max Queue Number: 0(0x0) 2022-03-17T19:03:38.1016601Z Queue Min Size: 0(0x0) 2022-03-17T19:03:38.1017300Z Queue Max Size: 0(0x0) 2022-03-17T19:03:38.1018003Z Queue Type: MULTI 2022-03-17T19:03:38.1018698Z Node: 0 2022-03-17T19:03:38.1019796Z Device Type: CPU 2022-03-17T19:03:38.1020455Z Cache Info: 2022-03-17T19:03:38.1021105Z L1: 32768(0x8000) KB 2022-03-17T19:03:38.1021771Z Chip ID: 0(0x0) 2022-03-17T19:03:38.1022480Z Cacheline Size: 64(0x40) 2022-03-17T19:03:38.1023193Z Max Clock Freq. (MHz): 2200 2022-03-17T19:03:38.1023875Z BDFID: 0 2022-03-17T19:03:38.1024560Z Internal Node ID: 0 2022-03-17T19:03:38.1025249Z Compute Unit: 16 2022-03-17T19:03:38.1025942Z SIMDs per CU: 0 2022-03-17T19:03:38.1026814Z Shader Engines: 0 2022-03-17T19:03:38.1027547Z Shader Arrs. per Eng.: 0 2022-03-17T19:03:38.1028281Z WatchPts on Addr. Ranges:1 2022-03-17T19:03:38.1028963Z Features: None 2022-03-17T19:03:38.1029567Z Pool Info: 2022-03-17T19:03:38.1030146Z Pool 1 2022-03-17T19:03:38.1030866Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-03-17T19:03:38.1031622Z Size: 65894480(0x3ed7850) KB 2022-03-17T19:03:38.1032343Z Allocatable: TRUE 2022-03-17T19:03:38.1033077Z Alloc Granule: 4KB 2022-03-17T19:03:38.1033794Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1034545Z Accessible by all: TRUE 2022-03-17T19:03:38.1035200Z Pool 2 2022-03-17T19:03:38.1035913Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-03-17T19:03:38.1036682Z Size: 65894480(0x3ed7850) KB 2022-03-17T19:03:38.1037384Z Allocatable: TRUE 2022-03-17T19:03:38.1038113Z Alloc Granule: 4KB 2022-03-17T19:03:38.1038820Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1039564Z Accessible by all: TRUE 2022-03-17T19:03:38.1040200Z ISA Info: 2022-03-17T19:03:38.1040772Z N/A 2022-03-17T19:03:38.1041323Z ******* 2022-03-17T19:03:38.1041973Z Agent 2 2022-03-17T19:03:38.1042532Z ******* 2022-03-17T19:03:38.1043464Z Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1044445Z Uuid: CPU-XX 2022-03-17T19:03:38.1045444Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1046237Z Vendor Name: CPU 2022-03-17T19:03:38.1046969Z Feature: None specified 2022-03-17T19:03:38.1047721Z Profile: FULL_PROFILE 2022-03-17T19:03:38.1048477Z Float Round Mode: NEAR 2022-03-17T19:03:38.1049190Z Max Queue Number: 0(0x0) 2022-03-17T19:03:38.1049907Z Queue Min Size: 0(0x0) 2022-03-17T19:03:38.1050614Z Queue Max Size: 0(0x0) 2022-03-17T19:03:38.1051321Z Queue Type: MULTI 2022-03-17T19:03:38.1052007Z Node: 1 2022-03-17T19:03:38.1052695Z Device Type: CPU 2022-03-17T19:03:38.1053348Z Cache Info: 2022-03-17T19:03:38.1054198Z L1: 32768(0x8000) KB 2022-03-17T19:03:38.1054868Z Chip ID: 0(0x0) 2022-03-17T19:03:38.1055561Z Cacheline Size: 64(0x40) 2022-03-17T19:03:38.1056267Z Max Clock Freq. (MHz): 2200 2022-03-17T19:03:38.1056955Z BDFID: 0 2022-03-17T19:03:38.1057650Z Internal Node ID: 1 2022-03-17T19:03:38.1058338Z Compute Unit: 16 2022-03-17T19:03:38.1059052Z SIMDs per CU: 0 2022-03-17T19:03:38.1059744Z Shader Engines: 0 2022-03-17T19:03:38.1060457Z Shader Arrs. per Eng.: 0 2022-03-17T19:03:38.1061319Z WatchPts on Addr. Ranges:1 2022-03-17T19:03:38.1062000Z Features: None 2022-03-17T19:03:38.1062614Z Pool Info: 2022-03-17T19:03:38.1063206Z Pool 1 2022-03-17T19:03:38.1063926Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-03-17T19:03:38.1064692Z Size: 66052532(0x3efe1b4) KB 2022-03-17T19:03:38.1065416Z Allocatable: TRUE 2022-03-17T19:03:38.1066149Z Alloc Granule: 4KB 2022-03-17T19:03:38.1066867Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1067616Z Accessible by all: TRUE 2022-03-17T19:03:38.1068259Z Pool 2 2022-03-17T19:03:38.1068983Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-03-17T19:03:38.1069765Z Size: 66052532(0x3efe1b4) KB 2022-03-17T19:03:38.1070475Z Allocatable: TRUE 2022-03-17T19:03:38.1071210Z Alloc Granule: 4KB 2022-03-17T19:03:38.1071931Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1072676Z Accessible by all: TRUE 2022-03-17T19:03:38.1073332Z ISA Info: 2022-03-17T19:03:38.1073904Z N/A 2022-03-17T19:03:38.1074446Z ******* 2022-03-17T19:03:38.1074976Z Agent 3 2022-03-17T19:03:38.1075516Z ******* 2022-03-17T19:03:38.1076439Z Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1077467Z Uuid: CPU-XX 2022-03-17T19:03:38.1078478Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1079304Z Vendor Name: CPU 2022-03-17T19:03:38.1080069Z Feature: None specified 2022-03-17T19:03:38.1080808Z Profile: FULL_PROFILE 2022-03-17T19:03:38.1081601Z Float Round Mode: NEAR 2022-03-17T19:03:38.1082692Z Max Queue Number: 0(0x0) 2022-03-17T19:03:38.1083396Z Queue Min Size: 0(0x0) 2022-03-17T19:03:38.1084098Z Queue Max Size: 0(0x0) 2022-03-17T19:03:38.1084794Z Queue Type: MULTI 2022-03-17T19:03:38.1085492Z Node: 2 2022-03-17T19:03:38.1086190Z Device Type: CPU 2022-03-17T19:03:38.1086840Z Cache Info: 2022-03-17T19:03:38.1087480Z L1: 32768(0x8000) KB 2022-03-17T19:03:38.1088152Z Chip ID: 0(0x0) 2022-03-17T19:03:38.1089048Z Cacheline Size: 64(0x40) 2022-03-17T19:03:38.1089765Z Max Clock Freq. (MHz): 2200 2022-03-17T19:03:38.1090450Z BDFID: 0 2022-03-17T19:03:38.1091124Z Internal Node ID: 2 2022-03-17T19:03:38.1091827Z Compute Unit: 16 2022-03-17T19:03:38.1092540Z SIMDs per CU: 0 2022-03-17T19:03:38.1093236Z Shader Engines: 0 2022-03-17T19:03:38.1093957Z Shader Arrs. per Eng.: 0 2022-03-17T19:03:38.1094693Z WatchPts on Addr. Ranges:1 2022-03-17T19:03:38.1095501Z Features: None 2022-03-17T19:03:38.1096087Z Pool Info: 2022-03-17T19:03:38.1096682Z Pool 1 2022-03-17T19:03:38.1097087Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-03-17T19:03:38.1097447Z Size: 66027676(0x3ef809c) KB 2022-03-17T19:03:38.1097731Z Allocatable: TRUE 2022-03-17T19:03:38.1098007Z Alloc Granule: 4KB 2022-03-17T19:03:38.1098281Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1098554Z Accessible by all: TRUE 2022-03-17T19:03:38.1098802Z Pool 2 2022-03-17T19:03:38.1099061Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-03-17T19:03:38.1099349Z Size: 66027676(0x3ef809c) KB 2022-03-17T19:03:38.1099621Z Allocatable: TRUE 2022-03-17T19:03:38.1099891Z Alloc Granule: 4KB 2022-03-17T19:03:38.1100166Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1100440Z Accessible by all: TRUE 2022-03-17T19:03:38.1100690Z ISA Info: 2022-03-17T19:03:38.1100906Z N/A 2022-03-17T19:03:38.1101102Z ******* 2022-03-17T19:03:38.1101314Z Agent 4 2022-03-17T19:03:38.1101515Z ******* 2022-03-17T19:03:38.1101874Z Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1102235Z Uuid: CPU-XX 2022-03-17T19:03:38.1102617Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:03:38.1102929Z Vendor Name: CPU 2022-03-17T19:03:38.1103203Z Feature: None specified 2022-03-17T19:03:38.1103499Z Profile: FULL_PROFILE 2022-03-17T19:03:38.1103781Z Float Round Mode: NEAR 2022-03-17T19:03:38.1104056Z Max Queue Number: 0(0x0) 2022-03-17T19:03:38.1104327Z Queue Min Size: 0(0x0) 2022-03-17T19:03:38.1104584Z Queue Max Size: 0(0x0) 2022-03-17T19:03:38.1104852Z Queue Type: MULTI 2022-03-17T19:03:38.1105104Z Node: 3 2022-03-17T19:03:38.1105464Z Device Type: CPU 2022-03-17T19:03:38.1105710Z Cache Info: 2022-03-17T19:03:38.1105956Z L1: 32768(0x8000) KB 2022-03-17T19:03:38.1106209Z Chip ID: 0(0x0) 2022-03-17T19:03:38.1106475Z Cacheline Size: 64(0x40) 2022-03-17T19:03:38.1106793Z Max Clock Freq. (MHz): 2200 2022-03-17T19:03:38.1107056Z BDFID: 0 2022-03-17T19:03:38.1107315Z Internal Node ID: 3 2022-03-17T19:03:38.1107573Z Compute Unit: 16 2022-03-17T19:03:38.1107838Z SIMDs per CU: 0 2022-03-17T19:03:38.1108094Z Shader Engines: 0 2022-03-17T19:03:38.1108365Z Shader Arrs. per Eng.: 0 2022-03-17T19:03:38.1108649Z WatchPts on Addr. Ranges:1 2022-03-17T19:03:38.1108896Z Features: None 2022-03-17T19:03:38.1109124Z Pool Info: 2022-03-17T19:03:38.1109340Z Pool 1 2022-03-17T19:03:38.1109657Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-03-17T19:03:38.1109945Z Size: 66051004(0x3efdbbc) KB 2022-03-17T19:03:38.1110229Z Allocatable: TRUE 2022-03-17T19:03:38.1110508Z Alloc Granule: 4KB 2022-03-17T19:03:38.1110777Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1111068Z Accessible by all: TRUE 2022-03-17T19:03:38.1111314Z Pool 2 2022-03-17T19:03:38.1111592Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-03-17T19:03:38.1111884Z Size: 66051004(0x3efdbbc) KB 2022-03-17T19:03:38.1112149Z Allocatable: TRUE 2022-03-17T19:03:38.1112425Z Alloc Granule: 4KB 2022-03-17T19:03:38.1112696Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1112981Z Accessible by all: TRUE 2022-03-17T19:03:38.1113234Z ISA Info: 2022-03-17T19:03:38.1113444Z N/A 2022-03-17T19:03:38.1113657Z ******* 2022-03-17T19:03:38.1113856Z Agent 5 2022-03-17T19:03:38.1114064Z ******* 2022-03-17T19:03:38.1114290Z Name: gfx906 2022-03-17T19:03:38.1114655Z Uuid: GPU-ebaa604172e620f6 2022-03-17T19:03:38.1114941Z Marketing Name: Vega 20 2022-03-17T19:03:38.1115205Z Vendor Name: AMD 2022-03-17T19:03:38.1115484Z Feature: KERNEL_DISPATCH 2022-03-17T19:03:38.1115765Z Profile: BASE_PROFILE 2022-03-17T19:03:38.1116054Z Float Round Mode: NEAR 2022-03-17T19:03:38.1116336Z Max Queue Number: 128(0x80) 2022-03-17T19:03:38.1116601Z Queue Min Size: 4096(0x1000) 2022-03-17T19:03:38.1116873Z Queue Max Size: 131072(0x20000) 2022-03-17T19:03:38.1117133Z Queue Type: MULTI 2022-03-17T19:03:38.1117393Z Node: 4 2022-03-17T19:03:38.1117647Z Device Type: GPU 2022-03-17T19:03:38.1117890Z Cache Info: 2022-03-17T19:03:38.1118137Z L1: 16(0x10) KB 2022-03-17T19:03:38.1118388Z Chip ID: 26273(0x66a1) 2022-03-17T19:03:38.1118656Z Cacheline Size: 64(0x40) 2022-03-17T19:03:38.1118919Z Max Clock Freq. (MHz): 1725 2022-03-17T19:03:38.1119180Z BDFID: 8960 2022-03-17T19:03:38.1119492Z Internal Node ID: 4 2022-03-17T19:03:38.1119755Z Compute Unit: 60 2022-03-17T19:03:38.1120022Z SIMDs per CU: 4 2022-03-17T19:03:38.1120280Z Shader Engines: 4 2022-03-17T19:03:38.1120552Z Shader Arrs. per Eng.: 1 2022-03-17T19:03:38.1120836Z WatchPts on Addr. Ranges:4 2022-03-17T19:03:38.1121103Z Features: KERNEL_DISPATCH 2022-03-17T19:03:38.1121386Z Fast F16 Operation: FALSE 2022-03-17T19:03:38.1121657Z Wavefront Size: 64(0x40) 2022-03-17T19:03:38.1122028Z Workgroup Max Size: 1024(0x400) 2022-03-17T19:03:38.1122375Z Workgroup Max Size per Dimension: 2022-03-17T19:03:38.1122643Z x 1024(0x400) 2022-03-17T19:03:38.1122893Z y 1024(0x400) 2022-03-17T19:03:38.1123129Z z 1024(0x400) 2022-03-17T19:03:38.1123388Z Max Waves Per CU: 40(0x28) 2022-03-17T19:03:38.1123764Z Max Work-item Per CU: 2560(0xa00) 2022-03-17T19:03:38.1124052Z Grid Max Size: 4294967295(0xffffffff) 2022-03-17T19:03:38.1124316Z Grid Max Size per Dimension: 2022-03-17T19:03:38.1124568Z x 4294967295(0xffffffff) 2022-03-17T19:03:38.1124832Z y 4294967295(0xffffffff) 2022-03-17T19:03:38.1125087Z z 4294967295(0xffffffff) 2022-03-17T19:03:38.1125365Z Max fbarriers/Workgrp: 32 2022-03-17T19:03:38.1125628Z Pool Info: 2022-03-17T19:03:38.1125846Z Pool 1 2022-03-17T19:03:38.1126122Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-03-17T19:03:38.1126403Z Size: 33538048(0x1ffc000) KB 2022-03-17T19:03:38.1126676Z Allocatable: TRUE 2022-03-17T19:03:38.1126945Z Alloc Granule: 4KB 2022-03-17T19:03:38.1127220Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1127504Z Accessible by all: FALSE 2022-03-17T19:03:38.1127747Z Pool 2 2022-03-17T19:03:38.1128000Z Segment: GROUP 2022-03-17T19:03:38.1128258Z Size: 64(0x40) KB 2022-03-17T19:03:38.1128528Z Allocatable: FALSE 2022-03-17T19:03:38.1128809Z Alloc Granule: 0KB 2022-03-17T19:03:38.1129081Z Alloc Alignment: 0KB 2022-03-17T19:03:38.1129371Z Accessible by all: FALSE 2022-03-17T19:03:38.1129614Z ISA Info: 2022-03-17T19:03:38.1129835Z ISA 1 2022-03-17T19:03:38.1130210Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-03-17T19:03:38.1130546Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-03-17T19:03:38.1130858Z Profiles: HSA_PROFILE_BASE 2022-03-17T19:03:38.1131151Z Default Rounding Mode: NEAR 2022-03-17T19:03:38.1131446Z Default Rounding Mode: NEAR 2022-03-17T19:03:38.1131718Z Fast f16: TRUE 2022-03-17T19:03:38.1131994Z Workgroup Max Size: 1024(0x400) 2022-03-17T19:03:38.1132273Z Workgroup Max Size per Dimension: 2022-03-17T19:03:38.1132597Z x 1024(0x400) 2022-03-17T19:03:38.1132858Z y 1024(0x400) 2022-03-17T19:03:38.1133101Z z 1024(0x400) 2022-03-17T19:03:38.1133375Z Grid Max Size: 4294967295(0xffffffff) 2022-03-17T19:03:38.1133649Z Grid Max Size per Dimension: 2022-03-17T19:03:38.1133907Z x 4294967295(0xffffffff) 2022-03-17T19:03:38.1134178Z y 4294967295(0xffffffff) 2022-03-17T19:03:38.1134445Z z 4294967295(0xffffffff) 2022-03-17T19:03:38.1134725Z FBarrier Max Size: 32 2022-03-17T19:03:38.1135006Z ******* 2022-03-17T19:03:38.1135208Z Agent 6 2022-03-17T19:03:38.1135415Z ******* 2022-03-17T19:03:38.1135645Z Name: gfx906 2022-03-17T19:03:38.1136018Z Uuid: GPU-3f2a88e172e620f4 2022-03-17T19:03:38.1136292Z Marketing Name: Vega 20 2022-03-17T19:03:38.1136569Z Vendor Name: AMD 2022-03-17T19:03:38.1136852Z Feature: KERNEL_DISPATCH 2022-03-17T19:03:38.1137132Z Profile: BASE_PROFILE 2022-03-17T19:03:38.1137415Z Float Round Mode: NEAR 2022-03-17T19:03:38.1137688Z Max Queue Number: 128(0x80) 2022-03-17T19:03:38.1137960Z Queue Min Size: 4096(0x1000) 2022-03-17T19:03:38.1138235Z Queue Max Size: 131072(0x20000) 2022-03-17T19:03:38.1138511Z Queue Type: MULTI 2022-03-17T19:03:38.1138800Z Node: 5 2022-03-17T19:03:38.1139070Z Device Type: GPU 2022-03-17T19:03:38.1139323Z Cache Info: 2022-03-17T19:03:38.1139576Z L1: 16(0x10) KB 2022-03-17T19:03:38.1139830Z Chip ID: 26273(0x66a1) 2022-03-17T19:03:38.1140105Z Cacheline Size: 64(0x40) 2022-03-17T19:03:38.1140373Z Max Clock Freq. (MHz): 1725 2022-03-17T19:03:38.1140641Z BDFID: 9728 2022-03-17T19:03:38.1140901Z Internal Node ID: 5 2022-03-17T19:03:38.1141175Z Compute Unit: 60 2022-03-17T19:03:38.1141455Z SIMDs per CU: 4 2022-03-17T19:03:38.1141725Z Shader Engines: 4 2022-03-17T19:03:38.1142013Z Shader Arrs. per Eng.: 1 2022-03-17T19:03:38.1142294Z WatchPts on Addr. Ranges:4 2022-03-17T19:03:38.1142576Z Features: KERNEL_DISPATCH 2022-03-17T19:03:38.1142866Z Fast F16 Operation: FALSE 2022-03-17T19:03:38.1143146Z Wavefront Size: 64(0x40) 2022-03-17T19:03:38.1143429Z Workgroup Max Size: 1024(0x400) 2022-03-17T19:03:38.1143699Z Workgroup Max Size per Dimension: 2022-03-17T19:03:38.1143969Z x 1024(0x400) 2022-03-17T19:03:38.1144223Z y 1024(0x400) 2022-03-17T19:03:38.1144466Z z 1024(0x400) 2022-03-17T19:03:38.1144736Z Max Waves Per CU: 40(0x28) 2022-03-17T19:03:38.1145158Z Max Work-item Per CU: 2560(0xa00) 2022-03-17T19:03:38.1145459Z Grid Max Size: 4294967295(0xffffffff) 2022-03-17T19:03:38.1145713Z Grid Max Size per Dimension: 2022-03-17T19:03:38.1145979Z x 4294967295(0xffffffff) 2022-03-17T19:03:38.1146244Z y 4294967295(0xffffffff) 2022-03-17T19:03:38.1146503Z z 4294967295(0xffffffff) 2022-03-17T19:03:38.1146781Z Max fbarriers/Workgrp: 32 2022-03-17T19:03:38.1147029Z Pool Info: 2022-03-17T19:03:38.1147256Z Pool 1 2022-03-17T19:03:38.1147533Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-03-17T19:03:38.1147814Z Size: 33538048(0x1ffc000) KB 2022-03-17T19:03:38.1148135Z Allocatable: TRUE 2022-03-17T19:03:38.1148408Z Alloc Granule: 4KB 2022-03-17T19:03:38.1148687Z Alloc Alignment: 4KB 2022-03-17T19:03:38.1148970Z Accessible by all: FALSE 2022-03-17T19:03:38.1149211Z Pool 2 2022-03-17T19:03:38.1149463Z Segment: GROUP 2022-03-17T19:03:38.1149725Z Size: 64(0x40) KB 2022-03-17T19:03:38.1149993Z Allocatable: FALSE 2022-03-17T19:03:38.1150263Z Alloc Granule: 0KB 2022-03-17T19:03:38.1150540Z Alloc Alignment: 0KB 2022-03-17T19:03:38.1150828Z Accessible by all: FALSE 2022-03-17T19:03:38.1151079Z ISA Info: 2022-03-17T19:03:38.1151297Z ISA 1 2022-03-17T19:03:38.1151680Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-03-17T19:03:38.1152012Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-03-17T19:03:38.1152320Z Profiles: HSA_PROFILE_BASE 2022-03-17T19:03:38.1152612Z Default Rounding Mode: NEAR 2022-03-17T19:03:38.1152908Z Default Rounding Mode: NEAR 2022-03-17T19:03:38.1153180Z Fast f16: TRUE 2022-03-17T19:03:38.1153451Z Workgroup Max Size: 1024(0x400) 2022-03-17T19:03:38.1153727Z Workgroup Max Size per Dimension: 2022-03-17T19:03:38.1153986Z x 1024(0x400) 2022-03-17T19:03:38.1154240Z y 1024(0x400) 2022-03-17T19:03:38.1154483Z z 1024(0x400) 2022-03-17T19:03:38.1154758Z Grid Max Size: 4294967295(0xffffffff) 2022-03-17T19:03:38.1155020Z Grid Max Size per Dimension: 2022-03-17T19:03:38.1155286Z x 4294967295(0xffffffff) 2022-03-17T19:03:38.1155558Z y 4294967295(0xffffffff) 2022-03-17T19:03:38.1155820Z z 4294967295(0xffffffff) 2022-03-17T19:03:38.1156095Z FBarrier Max Size: 32 2022-03-17T19:03:38.1156330Z *** Done *** 2022-03-17T19:03:38.1224015Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-03-17T19:03:38.1224946Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-03-17T19:03:38.1225727Z if [[ "x$ngpu" != "x2" && "x$ngpu" != "x4" ]]; then 2022-03-17T19:03:38.1226527Z  echo "Failed to detect GPUs on the runner" 2022-03-17T19:03:38.1227204Z  exit 1 2022-03-17T19:03:38.1227716Z fi 2022-03-17T19:03:38.1301100Z shell: /bin/bash -e {0} 2022-03-17T19:03:38.1301313Z env: 2022-03-17T19:03:38.1301599Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:38.1302047Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:38.1302480Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:38.1302870Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:38.1303195Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:38.1303443Z IN_CI: 1 2022-03-17T19:03:38.1303635Z IS_GHA: 1 2022-03-17T19:03:38.1303844Z IN_WHEEL_TEST: 1 2022-03-17T19:03:38.1304144Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:38.1304503Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:38.1304909Z PR_LABELS: [] 2022-03-17T19:03:38.1305346Z GITHUB_TOKEN: *** 2022-03-17T19:03:38.1305589Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:38.1305829Z PR_NUMBER: 2022-03-17T19:03:38.1306066Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:38.1306333Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:38.1306576Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:38.1307014Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:38.1307505Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:38.1307801Z TEST_CONFIG: distributed 2022-03-17T19:03:38.1308036Z SHARD_NUMBER: 1 2022-03-17T19:03:38.1308245Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:38.1308457Z PR_BODY: 2022-03-17T19:03:38.1308703Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:03:38.1308955Z ##[endgroup] 2022-03-17T19:03:38.2212589Z ##[group]Run env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}" 2022-03-17T19:03:38.2213042Z env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}" 2022-03-17T19:03:38.2246653Z shell: /bin/bash -e {0} 2022-03-17T19:03:38.2246924Z env: 2022-03-17T19:03:38.2247262Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:38.2247812Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:38.2248334Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:38.2248785Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:38.2249191Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:38.2249448Z IN_CI: 1 2022-03-17T19:03:38.2249678Z IS_GHA: 1 2022-03-17T19:03:38.2249916Z IN_WHEEL_TEST: 1 2022-03-17T19:03:38.2250258Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:38.2250698Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:38.2251068Z PR_LABELS: [] 2022-03-17T19:03:38.2251566Z GITHUB_TOKEN: *** 2022-03-17T19:03:38.2251857Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:38.2252141Z PR_NUMBER: 2022-03-17T19:03:38.2252423Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:38.2252740Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:38.2253041Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:38.2253548Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:38.2254143Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:38.2254508Z TEST_CONFIG: distributed 2022-03-17T19:03:38.2254772Z SHARD_NUMBER: 1 2022-03-17T19:03:38.2255034Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:38.2255283Z PR_BODY: 2022-03-17T19:03:38.2255586Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:03:38.2255875Z ##[endgroup] 2022-03-17T19:03:38.2543463Z ##[group]Run zhouzhuojie/checkout@05b13c9a0d21f08f6d5e64a1d5042246d13619d9 2022-03-17T19:03:38.2543767Z with: 2022-03-17T19:03:38.2544003Z ref: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:38.2544238Z fetch-depth: 0 2022-03-17T19:03:38.2544459Z submodules: recursive 2022-03-17T19:03:38.2544701Z repository: pytorch/pytorch 2022-03-17T19:03:38.2545103Z token: *** 2022-03-17T19:03:38.2545310Z ssh-strict: true 2022-03-17T19:03:38.2545538Z persist-credentials: true 2022-03-17T19:03:38.2545771Z clean: true 2022-03-17T19:03:38.2545959Z lfs: false 2022-03-17T19:03:38.2546151Z env: 2022-03-17T19:03:38.2546432Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:03:38.2546870Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:03:38.2547303Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:03:38.2547707Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:03:38.2548167Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:03:38.2548521Z IN_CI: 1 2022-03-17T19:03:38.2548765Z IS_GHA: 1 2022-03-17T19:03:38.2548961Z IN_WHEEL_TEST: 1 2022-03-17T19:03:38.2549244Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:03:38.2549589Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:03:38.2549883Z PR_LABELS: [] 2022-03-17T19:03:38.2550222Z GITHUB_TOKEN: *** 2022-03-17T19:03:38.2550454Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:03:38.2550689Z PR_NUMBER: 2022-03-17T19:03:38.2550924Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:03:38.2551183Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:03:38.2551427Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:03:38.2551859Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:03:38.2552355Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:03:38.2552659Z TEST_CONFIG: distributed 2022-03-17T19:03:38.2552886Z SHARD_NUMBER: 1 2022-03-17T19:03:38.2553097Z NUM_TEST_SHARDS: 1 2022-03-17T19:03:38.2553331Z PR_BODY: 2022-03-17T19:03:38.2553590Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:03:38.2553839Z ##[endgroup] 2022-03-17T19:03:38.3866406Z Syncing repository: pytorch/pytorch 2022-03-17T19:03:38.3867906Z ##[group]Getting Git version info 2022-03-17T19:03:38.3868499Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-03-17T19:03:38.3869030Z [command]/usr/bin/git version 2022-03-17T19:03:38.3869259Z git version 2.35.1 2022-03-17T19:03:38.3869918Z ##[endgroup] 2022-03-17T19:03:38.3870340Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-03-17T19:03:38.3871472Z ##[group]Initializing the repository 2022-03-17T19:03:38.3871935Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-03-17T19:03:38.3872480Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-03-17T19:03:38.3872960Z hint: is subject to change. To configure the initial branch name to use in all 2022-03-17T19:03:38.3873427Z hint: of your new repositories, which will suppress this warning, call: 2022-03-17T19:03:38.3873765Z hint: 2022-03-17T19:03:38.3874189Z hint: git config --global init.defaultBranch 2022-03-17T19:03:38.3874494Z hint: 2022-03-17T19:03:38.3874947Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-03-17T19:03:38.3875505Z hint: 'development'. The just-created branch can be renamed via this command: 2022-03-17T19:03:38.3875853Z hint: 2022-03-17T19:03:38.3876173Z hint: git branch -m 2022-03-17T19:03:38.3876728Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2022-03-17T19:03:38.3877282Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-03-17T19:03:38.3898707Z ##[endgroup] 2022-03-17T19:03:38.3899306Z ##[group]Disabling automatic garbage collection 2022-03-17T19:03:38.3909370Z [command]/usr/bin/git config --local gc.auto 0 2022-03-17T19:03:38.3958777Z ##[endgroup] 2022-03-17T19:03:38.4000554Z ##[group]Setting up auth 2022-03-17T19:03:38.4002113Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-03-17T19:03:38.4052369Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-03-17T19:03:38.4366183Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-03-17T19:03:38.4424372Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-03-17T19:03:38.4774289Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-03-17T19:03:38.4857570Z ##[endgroup] 2022-03-17T19:03:38.4858704Z ##[group]Fetching the repository 2022-03-17T19:03:38.4873356Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-03-17T19:03:42.7675727Z remote: Enumerating objects: 767821 2022-03-17T19:03:42.7676551Z remote: Enumerating objects: 768219, done. 2022-03-17T19:03:42.7680818Z remote: Counting objects: 0% (1/398) 2022-03-17T19:03:42.7681554Z remote: Counting objects: 1% (4/398) 2022-03-17T19:03:42.7682412Z remote: Counting objects: 2% (8/398) 2022-03-17T19:03:42.7683128Z remote: Counting objects: 3% (12/398) 2022-03-17T19:03:42.7683840Z remote: Counting objects: 4% (16/398) 2022-03-17T19:03:42.7684565Z remote: Counting objects: 5% (20/398) 2022-03-17T19:03:42.7685309Z remote: Counting objects: 6% (24/398) 2022-03-17T19:03:42.7686022Z remote: Counting objects: 7% (28/398) 2022-03-17T19:03:42.7686730Z remote: Counting objects: 8% (32/398) 2022-03-17T19:03:42.7687437Z remote: Counting objects: 9% (36/398) 2022-03-17T19:03:42.7691410Z remote: Counting objects: 10% (40/398) 2022-03-17T19:03:42.7692500Z remote: Counting objects: 11% (44/398) 2022-03-17T19:03:42.7693525Z remote: Counting objects: 12% (48/398) 2022-03-17T19:03:42.7694458Z remote: Counting objects: 13% (52/398) 2022-03-17T19:03:42.7695670Z remote: Counting objects: 14% (56/398) 2022-03-17T19:03:42.7696644Z remote: Counting objects: 15% (60/398) 2022-03-17T19:03:42.7697576Z remote: Counting objects: 16% (64/398) 2022-03-17T19:03:42.7698487Z remote: Counting objects: 17% (68/398) 2022-03-17T19:03:42.7699358Z remote: Counting objects: 18% (72/398) 2022-03-17T19:03:42.7700071Z remote: Counting objects: 19% (76/398) 2022-03-17T19:03:42.7700845Z remote: Counting objects: 20% (80/398) 2022-03-17T19:03:42.7701592Z remote: Counting objects: 21% (84/398) 2022-03-17T19:03:42.7702324Z remote: Counting objects: 22% (88/398) 2022-03-17T19:03:42.7703064Z remote: Counting objects: 23% (92/398) 2022-03-17T19:03:42.7703788Z remote: Counting objects: 24% (96/398) 2022-03-17T19:03:42.7704525Z remote: Counting objects: 25% (100/398) 2022-03-17T19:03:42.7705255Z remote: Counting objects: 26% (104/398) 2022-03-17T19:03:42.7705991Z remote: Counting objects: 27% (108/398) 2022-03-17T19:03:42.7706724Z remote: Counting objects: 28% (112/398) 2022-03-17T19:03:42.7707432Z remote: Counting objects: 29% (116/398) 2022-03-17T19:03:42.7708159Z remote: Counting objects: 30% (120/398) 2022-03-17T19:03:42.7708883Z remote: Counting objects: 31% (124/398) 2022-03-17T19:03:42.7709616Z remote: Counting objects: 32% (128/398) 2022-03-17T19:03:42.7710937Z remote: Counting objects: 33% (132/398) 2022-03-17T19:03:42.7711671Z remote: Counting objects: 34% (136/398) 2022-03-17T19:03:42.7712406Z remote: Counting objects: 35% (140/398) 2022-03-17T19:03:42.7713124Z remote: Counting objects: 36% (144/398) 2022-03-17T19:03:42.7713859Z remote: Counting objects: 37% (148/398) 2022-03-17T19:03:42.7714572Z remote: Counting objects: 38% (152/398) 2022-03-17T19:03:42.7715323Z remote: Counting objects: 39% (156/398) 2022-03-17T19:03:42.7716124Z remote: Counting objects: 40% (160/398) 2022-03-17T19:03:42.7716913Z remote: Counting objects: 41% (164/398) 2022-03-17T19:03:42.7717629Z remote: Counting objects: 42% (168/398) 2022-03-17T19:03:42.7718325Z remote: Counting objects: 43% (172/398) 2022-03-17T19:03:42.7719033Z remote: Counting objects: 44% (176/398) 2022-03-17T19:03:42.7719747Z remote: Counting objects: 45% (180/398) 2022-03-17T19:03:42.7720638Z remote: Counting objects: 46% (184/398) 2022-03-17T19:03:42.7721350Z remote: Counting objects: 47% (188/398) 2022-03-17T19:03:42.7722226Z remote: Counting objects: 48% (192/398) 2022-03-17T19:03:42.7722950Z remote: Counting objects: 49% (196/398) 2022-03-17T19:03:42.7723647Z remote: Counting objects: 50% (199/398) 2022-03-17T19:03:42.7724356Z remote: Counting objects: 51% (203/398) 2022-03-17T19:03:42.7725071Z remote: Counting objects: 52% (207/398) 2022-03-17T19:03:42.7725767Z remote: Counting objects: 53% (211/398) 2022-03-17T19:03:42.7726479Z remote: Counting objects: 54% (215/398) 2022-03-17T19:03:42.7727176Z remote: Counting objects: 55% (219/398) 2022-03-17T19:03:42.7727885Z remote: Counting objects: 56% (223/398) 2022-03-17T19:03:42.7728598Z remote: Counting objects: 57% (227/398) 2022-03-17T19:03:42.7729294Z remote: Counting objects: 58% (231/398) 2022-03-17T19:03:42.7730007Z remote: Counting objects: 59% (235/398) 2022-03-17T19:03:42.7730706Z remote: Counting objects: 60% (239/398) 2022-03-17T19:03:42.7731416Z remote: Counting objects: 61% (243/398) 2022-03-17T19:03:42.7732110Z remote: Counting objects: 62% (247/398) 2022-03-17T19:03:42.7732817Z remote: Counting objects: 63% (251/398) 2022-03-17T19:03:42.7733527Z remote: Counting objects: 64% (255/398) 2022-03-17T19:03:42.7734221Z remote: Counting objects: 65% (259/398) 2022-03-17T19:03:42.7734931Z remote: Counting objects: 66% (263/398) 2022-03-17T19:03:42.7735622Z remote: Counting objects: 67% (267/398) 2022-03-17T19:03:42.7736336Z remote: Counting objects: 68% (271/398) 2022-03-17T19:03:42.7737044Z remote: Counting objects: 69% (275/398) 2022-03-17T19:03:42.7737737Z remote: Counting objects: 70% (279/398) 2022-03-17T19:03:42.7738442Z remote: Counting objects: 71% (283/398) 2022-03-17T19:03:42.7739146Z remote: Counting objects: 72% (287/398) 2022-03-17T19:03:42.7739864Z remote: Counting objects: 73% (291/398) 2022-03-17T19:03:42.7740561Z remote: Counting objects: 74% (295/398) 2022-03-17T19:03:42.7741274Z remote: Counting objects: 75% (299/398) 2022-03-17T19:03:42.7741989Z remote: Counting objects: 76% (303/398) 2022-03-17T19:03:42.7742684Z remote: Counting objects: 77% (307/398) 2022-03-17T19:03:42.7743400Z remote: Counting objects: 78% (311/398) 2022-03-17T19:03:42.7744104Z remote: Counting objects: 79% (315/398) 2022-03-17T19:03:42.7744813Z remote: Counting objects: 80% (319/398) 2022-03-17T19:03:42.7745521Z remote: Counting objects: 81% (323/398) 2022-03-17T19:03:42.7746214Z remote: Counting objects: 82% (327/398) 2022-03-17T19:03:42.7746923Z remote: Counting objects: 83% (331/398) 2022-03-17T19:03:42.7747658Z remote: Counting objects: 84% (335/398) 2022-03-17T19:03:42.7748525Z remote: Counting objects: 85% (339/398) 2022-03-17T19:03:42.7749263Z remote: Counting objects: 86% (343/398) 2022-03-17T19:03:42.7749960Z remote: Counting objects: 87% (347/398) 2022-03-17T19:03:42.7750671Z remote: Counting objects: 88% (351/398) 2022-03-17T19:03:42.7751372Z remote: Counting objects: 89% (355/398) 2022-03-17T19:03:42.7752093Z remote: Counting objects: 90% (359/398) 2022-03-17T19:03:42.7752794Z remote: Counting objects: 91% (363/398) 2022-03-17T19:03:42.7753510Z remote: Counting objects: 92% (367/398) 2022-03-17T19:03:42.7754226Z remote: Counting objects: 93% (371/398) 2022-03-17T19:03:42.7754929Z remote: Counting objects: 94% (375/398) 2022-03-17T19:03:42.7755642Z remote: Counting objects: 95% (379/398) 2022-03-17T19:03:42.7756345Z remote: Counting objects: 96% (383/398) 2022-03-17T19:03:42.7757058Z remote: Counting objects: 97% (387/398) 2022-03-17T19:03:42.7757785Z remote: Counting objects: 98% (391/398) 2022-03-17T19:03:42.7758610Z remote: Counting objects: 99% (395/398) 2022-03-17T19:03:42.7759326Z remote: Counting objects: 100% (398/398) 2022-03-17T19:03:42.7760054Z remote: Counting objects: 100% (398/398), done. 2022-03-17T19:03:42.7760837Z remote: Compressing objects: 0% (1/295) 2022-03-17T19:03:42.7761595Z remote: Compressing objects: 1% (3/295) 2022-03-17T19:03:42.7762687Z remote: Compressing objects: 2% (6/295) 2022-03-17T19:03:42.7790458Z remote: Compressing objects: 3% (9/295) 2022-03-17T19:03:42.7902612Z remote: Compressing objects: 4% (12/295) 2022-03-17T19:03:42.7903452Z remote: Compressing objects: 5% (15/295) 2022-03-17T19:03:42.8081144Z remote: Compressing objects: 6% (18/295) 2022-03-17T19:03:42.8158159Z remote: Compressing objects: 7% (21/295) 2022-03-17T19:03:42.8270918Z remote: Compressing objects: 8% (24/295) 2022-03-17T19:03:42.8364818Z remote: Compressing objects: 9% (27/295) 2022-03-17T19:03:42.8370851Z remote: Compressing objects: 10% (30/295) 2022-03-17T19:03:42.8371661Z remote: Compressing objects: 11% (33/295) 2022-03-17T19:03:42.8372450Z remote: Compressing objects: 12% (36/295) 2022-03-17T19:03:42.8373219Z remote: Compressing objects: 13% (39/295) 2022-03-17T19:03:42.8373994Z remote: Compressing objects: 14% (42/295) 2022-03-17T19:03:42.8375012Z remote: Compressing objects: 15% (45/295) 2022-03-17T19:03:42.8375788Z remote: Compressing objects: 16% (48/295) 2022-03-17T19:03:42.8378851Z remote: Compressing objects: 17% (51/295) 2022-03-17T19:03:42.8383353Z remote: Compressing objects: 18% (54/295) 2022-03-17T19:03:42.8384158Z remote: Compressing objects: 19% (57/295) 2022-03-17T19:03:42.8387740Z remote: Compressing objects: 20% (59/295) 2022-03-17T19:03:42.8388531Z remote: Compressing objects: 21% (62/295) 2022-03-17T19:03:42.8392034Z remote: Compressing objects: 22% (65/295) 2022-03-17T19:03:42.8393931Z remote: Compressing objects: 23% (68/295) 2022-03-17T19:03:42.8394737Z remote: Compressing objects: 24% (71/295) 2022-03-17T19:03:42.8395500Z remote: Compressing objects: 25% (74/295) 2022-03-17T19:03:42.8396278Z remote: Compressing objects: 26% (77/295) 2022-03-17T19:03:42.8397039Z remote: Compressing objects: 27% (80/295) 2022-03-17T19:03:42.8397808Z remote: Compressing objects: 28% (83/295) 2022-03-17T19:03:42.8398598Z remote: Compressing objects: 29% (86/295) 2022-03-17T19:03:42.8399356Z remote: Compressing objects: 30% (89/295) 2022-03-17T19:03:42.8400140Z remote: Compressing objects: 31% (92/295) 2022-03-17T19:03:42.8402413Z remote: Compressing objects: 32% (95/295) 2022-03-17T19:03:42.8405435Z remote: Compressing objects: 33% (98/295) 2022-03-17T19:03:42.8411378Z remote: Compressing objects: 34% (101/295) 2022-03-17T19:03:42.8412446Z remote: Compressing objects: 35% (104/295) 2022-03-17T19:03:42.8413262Z remote: Compressing objects: 36% (107/295) 2022-03-17T19:03:42.8414042Z remote: Compressing objects: 37% (110/295) 2022-03-17T19:03:42.8416515Z remote: Compressing objects: 38% (113/295) 2022-03-17T19:03:42.8417314Z remote: Compressing objects: 39% (116/295) 2022-03-17T19:03:42.8418090Z remote: Compressing objects: 40% (118/295) 2022-03-17T19:03:42.8418854Z remote: Compressing objects: 41% (121/295) 2022-03-17T19:03:42.8419629Z remote: Compressing objects: 42% (124/295) 2022-03-17T19:03:42.8420765Z remote: Compressing objects: 43% (127/295) 2022-03-17T19:03:42.8421583Z remote: Compressing objects: 44% (130/295) 2022-03-17T19:03:42.8422354Z remote: Compressing objects: 45% (133/295) 2022-03-17T19:03:42.8423124Z remote: Compressing objects: 46% (136/295) 2022-03-17T19:03:42.8423925Z remote: Compressing objects: 47% (139/295) 2022-03-17T19:03:42.8424867Z remote: Compressing objects: 48% (142/295) 2022-03-17T19:03:42.8425645Z remote: Compressing objects: 49% (145/295) 2022-03-17T19:03:42.8426407Z remote: Compressing objects: 50% (148/295) 2022-03-17T19:03:42.8427208Z remote: Compressing objects: 51% (151/295) 2022-03-17T19:03:42.8428207Z remote: Compressing objects: 52% (154/295) 2022-03-17T19:03:42.8433137Z remote: Compressing objects: 53% (157/295) 2022-03-17T19:03:42.8449395Z remote: Compressing objects: 54% (160/295) 2022-03-17T19:03:42.8450246Z remote: Compressing objects: 55% (163/295) 2022-03-17T19:03:42.8451022Z remote: Compressing objects: 56% (166/295) 2022-03-17T19:03:42.8451817Z remote: Compressing objects: 57% (169/295) 2022-03-17T19:03:42.8459350Z remote: Compressing objects: 58% (172/295) 2022-03-17T19:03:42.8463354Z remote: Compressing objects: 59% (175/295) 2022-03-17T19:03:42.8464280Z remote: Compressing objects: 60% (177/295) 2022-03-17T19:03:42.8465086Z remote: Compressing objects: 61% (180/295) 2022-03-17T19:03:42.8465862Z remote: Compressing objects: 62% (183/295) 2022-03-17T19:03:42.8466636Z remote: Compressing objects: 63% (186/295) 2022-03-17T19:03:42.8467451Z remote: Compressing objects: 64% (189/295) 2022-03-17T19:03:42.8468225Z remote: Compressing objects: 65% (192/295) 2022-03-17T19:03:42.8471946Z remote: Compressing objects: 66% (195/295) 2022-03-17T19:03:42.8477780Z remote: Compressing objects: 67% (198/295) 2022-03-17T19:03:42.8479225Z remote: Compressing objects: 68% (201/295) 2022-03-17T19:03:42.8483805Z remote: Compressing objects: 69% (204/295) 2022-03-17T19:03:42.8484357Z remote: Compressing objects: 70% (207/295) 2022-03-17T19:03:42.8488073Z remote: Compressing objects: 71% (210/295) 2022-03-17T19:03:42.8488623Z remote: Compressing objects: 72% (213/295) 2022-03-17T19:03:42.8500804Z remote: Compressing objects: 73% (216/295) 2022-03-17T19:03:42.8501565Z remote: Compressing objects: 74% (219/295) 2022-03-17T19:03:42.8502291Z remote: Compressing objects: 75% (222/295) 2022-03-17T19:03:42.8502951Z remote: Compressing objects: 76% (225/295) 2022-03-17T19:03:42.8503512Z remote: Compressing objects: 77% (228/295) 2022-03-17T19:03:42.8504077Z remote: Compressing objects: 78% (231/295) 2022-03-17T19:03:42.8504720Z remote: Compressing objects: 79% (234/295) 2022-03-17T19:03:42.8505295Z remote: Compressing objects: 80% (236/295) 2022-03-17T19:03:42.8505849Z remote: Compressing objects: 81% (239/295) 2022-03-17T19:03:42.8506472Z remote: Compressing objects: 82% (242/295) 2022-03-17T19:03:42.8507649Z remote: Compressing objects: 83% (245/295) 2022-03-17T19:03:42.8508466Z remote: Compressing objects: 84% (248/295) 2022-03-17T19:03:42.8509442Z remote: Compressing objects: 85% (251/295) 2022-03-17T19:03:42.8510098Z remote: Compressing objects: 86% (254/295) 2022-03-17T19:03:42.8510659Z remote: Compressing objects: 87% (257/295) 2022-03-17T19:03:42.8514198Z remote: Compressing objects: 88% (260/295) 2022-03-17T19:03:42.8514914Z remote: Compressing objects: 89% (263/295) 2022-03-17T19:03:42.8515616Z remote: Compressing objects: 90% (266/295) 2022-03-17T19:03:42.8518150Z remote: Compressing objects: 91% (269/295) 2022-03-17T19:03:42.8518834Z remote: Compressing objects: 92% (272/295) 2022-03-17T19:03:42.8522063Z remote: Compressing objects: 93% (275/295) 2022-03-17T19:03:42.8522754Z remote: Compressing objects: 94% (278/295) 2022-03-17T19:03:42.8523626Z remote: Compressing objects: 95% (281/295) 2022-03-17T19:03:42.8524299Z remote: Compressing objects: 96% (284/295) 2022-03-17T19:03:42.8524973Z remote: Compressing objects: 97% (287/295) 2022-03-17T19:03:42.8525912Z remote: Compressing objects: 98% (290/295) 2022-03-17T19:03:42.8526795Z remote: Compressing objects: 99% (293/295) 2022-03-17T19:03:42.8527485Z remote: Compressing objects: 100% (295/295) 2022-03-17T19:03:42.8528081Z remote: Compressing objects: 100% (295/295), done. 2022-03-17T19:03:43.8942096Z Receiving objects: 0% (1/768219) 2022-03-17T19:03:44.1975544Z Receiving objects: 0% (4790/768219), 636.01 KiB | 1.13 MiB/s 2022-03-17T19:03:44.8802719Z Receiving objects: 1% (7683/768219), 1.55 MiB | 1.47 MiB/s 2022-03-17T19:03:45.0173451Z Receiving objects: 1% (13217/768219), 2.61 MiB | 1.66 MiB/s 2022-03-17T19:03:45.8720784Z Receiving objects: 2% (15365/768219), 3.96 MiB | 1.87 MiB/s 2022-03-17T19:03:45.8728646Z Receiving objects: 2% (23011/768219), 5.38 MiB | 2.06 MiB/s 2022-03-17T19:03:46.5372862Z Receiving objects: 3% (23047/768219), 5.38 MiB | 2.06 MiB/s 2022-03-17T19:03:46.8915154Z Receiving objects: 4% (30729/768219), 9.07 MiB | 2.50 MiB/s 2022-03-17T19:03:47.1730581Z Receiving objects: 4% (34567/768219), 9.07 MiB | 2.50 MiB/s 2022-03-17T19:03:47.8294211Z Receiving objects: 5% (38411/768219), 11.08 MiB | 2.68 MiB/s 2022-03-17T19:03:47.8704966Z Receiving objects: 6% (46094/768219), 13.30 MiB | 2.86 MiB/s 2022-03-17T19:03:48.4142422Z Receiving objects: 6% (47165/768219), 13.30 MiB | 2.86 MiB/s 2022-03-17T19:03:48.8722677Z Receiving objects: 7% (53776/768219), 15.48 MiB | 3.23 MiB/s 2022-03-17T19:03:48.9657335Z Receiving objects: 7% (59208/768219), 17.87 MiB | 3.54 MiB/s 2022-03-17T19:03:49.5059881Z Receiving objects: 8% (61458/768219), 17.87 MiB | 3.54 MiB/s 2022-03-17T19:03:49.8884635Z Receiving objects: 9% (69140/768219), 20.14 MiB | 3.82 MiB/s 2022-03-17T19:03:50.0598542Z Receiving objects: 9% (74438/768219), 22.63 MiB | 4.09 MiB/s 2022-03-17T19:03:50.7659647Z Receiving objects: 10% (76822/768219), 24.95 MiB | 4.29 MiB/s 2022-03-17T19:03:50.8774773Z Receiving objects: 11% (84505/768219), 27.45 MiB | 4.43 MiB/s 2022-03-17T19:03:51.2349975Z Receiving objects: 11% (85879/768219), 27.45 MiB | 4.43 MiB/s 2022-03-17T19:03:51.8704238Z Receiving objects: 12% (92187/768219), 29.75 MiB | 4.53 MiB/s 2022-03-17T19:03:52.6473986Z Receiving objects: 12% (96177/768219), 32.29 MiB | 4.63 MiB/s 2022-03-17T19:03:52.8713722Z Receiving objects: 13% (99869/768219), 37.12 MiB | 4.73 MiB/s 2022-03-17T19:03:53.8878887Z Receiving objects: 13% (101006/768219), 37.12 MiB | 4.73 MiB/s 2022-03-17T19:03:54.6333428Z Receiving objects: 13% (106060/768219), 42.00 MiB | 4.77 MiB/s 2022-03-17T19:03:54.9059383Z Receiving objects: 14% (107551/768219), 46.97 MiB | 4.81 MiB/s 2022-03-17T19:03:55.8706174Z Receiving objects: 14% (108157/768219), 46.97 MiB | 4.81 MiB/s 2022-03-17T19:03:56.8804126Z Receiving objects: 14% (110822/768219), 52.17 MiB | 4.90 MiB/s 2022-03-17T19:03:56.8990321Z Receiving objects: 14% (114883/768219), 57.66 MiB | 5.03 MiB/s 2022-03-17T19:03:57.8442018Z Receiving objects: 15% (115233/768219), 57.66 MiB | 5.03 MiB/s 2022-03-17T19:03:57.8965165Z Receiving objects: 16% (122916/768219), 63.91 MiB | 5.33 MiB/s 2022-03-17T19:03:58.8733328Z Receiving objects: 16% (123029/768219), 63.91 MiB | 5.33 MiB/s 2022-03-17T19:03:59.3246931Z Receiving objects: 16% (128792/768219), 70.92 MiB | 5.78 MiB/s 2022-03-17T19:03:59.8706478Z Receiving objects: 17% (130598/768219), 74.64 MiB | 6.03 MiB/s 2022-03-17T19:04:00.9065186Z Receiving objects: 17% (132833/768219), 78.96 MiB | 6.42 MiB/s 2022-03-17T19:04:01.0223085Z Receiving objects: 17% (137386/768219), 88.42 MiB | 7.35 MiB/s 2022-03-17T19:04:01.9130313Z Receiving objects: 18% (138280/768219), 88.42 MiB | 7.35 MiB/s 2022-03-17T19:04:02.1686568Z Receiving objects: 18% (144222/768219), 99.05 MiB | 8.45 MiB/s 2022-03-17T19:04:02.8708975Z Receiving objects: 19% (145962/768219), 99.05 MiB | 8.45 MiB/s 2022-03-17T19:04:02.9380651Z Receiving objects: 19% (153099/768219), 110.99 MiB | 9.66 MiB/s 2022-03-17T19:04:03.6499651Z Receiving objects: 20% (153644/768219), 110.99 MiB | 9.66 MiB/s 2022-03-17T19:04:03.8731459Z Receiving objects: 21% (161326/768219), 118.11 MiB | 10.40 MiB/s 2022-03-17T19:04:04.2109253Z Receiving objects: 21% (164257/768219), 126.00 MiB | 11.34 MiB/s 2022-03-17T19:04:04.6490933Z Receiving objects: 22% (169009/768219), 126.00 MiB | 11.34 MiB/s 2022-03-17T19:04:04.8704882Z Receiving objects: 23% (176691/768219), 134.56 MiB | 12.28 MiB/s 2022-03-17T19:04:05.0293900Z Receiving objects: 23% (181134/768219), 143.50 MiB | 13.33 MiB/s 2022-03-17T19:04:05.3302961Z Receiving objects: 24% (184373/768219), 143.50 MiB | 13.33 MiB/s 2022-03-17T19:04:05.7122843Z Receiving objects: 25% (192055/768219), 153.18 MiB | 14.37 MiB/s 2022-03-17T19:04:05.8704468Z Receiving objects: 26% (199737/768219), 153.18 MiB | 14.37 MiB/s 2022-03-17T19:04:06.2050658Z Receiving objects: 26% (202861/768219), 160.85 MiB | 14.94 MiB/s 2022-03-17T19:04:06.6475922Z Receiving objects: 27% (207420/768219), 160.85 MiB | 14.94 MiB/s 2022-03-17T19:04:06.8708072Z Receiving objects: 28% (215102/768219), 168.47 MiB | 15.40 MiB/s 2022-03-17T19:04:07.0383655Z Receiving objects: 28% (219091/768219), 177.39 MiB | 16.19 MiB/s 2022-03-17T19:04:07.4361941Z Receiving objects: 29% (222784/768219), 177.39 MiB | 16.19 MiB/s 2022-03-17T19:04:07.8712180Z Receiving objects: 30% (230466/768219), 186.11 MiB | 16.67 MiB/s 2022-03-17T19:04:07.8764701Z Receiving objects: 30% (238034/768219), 195.72 MiB | 17.24 MiB/s 2022-03-17T19:04:08.2340806Z Receiving objects: 31% (238148/768219), 195.72 MiB | 17.24 MiB/s 2022-03-17T19:04:08.5977840Z Receiving objects: 32% (245831/768219), 195.72 MiB | 17.24 MiB/s 2022-03-17T19:04:08.8704606Z Receiving objects: 33% (253513/768219), 205.38 MiB | 17.63 MiB/s 2022-03-17T19:04:08.9209127Z Receiving objects: 33% (260366/768219), 213.48 MiB | 17.53 MiB/s 2022-03-17T19:04:09.5713228Z Receiving objects: 34% (261195/768219), 213.48 MiB | 17.53 MiB/s 2022-03-17T19:04:09.8180348Z Receiving objects: 35% (268877/768219), 221.22 MiB | 17.26 MiB/s 2022-03-17T19:04:09.8709665Z Receiving objects: 36% (276559/768219), 232.00 MiB | 17.51 MiB/s 2022-03-17T19:04:10.0534995Z Receiving objects: 36% (277703/768219), 232.00 MiB | 17.51 MiB/s 2022-03-17T19:04:10.2999575Z Receiving objects: 37% (284242/768219), 232.00 MiB | 17.51 MiB/s 2022-03-17T19:04:10.6681198Z Receiving objects: 38% (291924/768219), 242.95 MiB | 18.24 MiB/s 2022-03-17T19:04:10.8720407Z Receiving objects: 39% (299606/768219), 242.95 MiB | 18.24 MiB/s 2022-03-17T19:04:11.0569532Z Receiving objects: 39% (305680/768219), 253.21 MiB | 18.83 MiB/s 2022-03-17T19:04:11.3969087Z Receiving objects: 40% (307288/768219), 253.21 MiB | 18.83 MiB/s 2022-03-17T19:04:11.7706658Z Receiving objects: 41% (314970/768219), 261.29 MiB | 18.63 MiB/s 2022-03-17T19:04:11.8702639Z Receiving objects: 42% (322652/768219), 269.21 MiB | 18.45 MiB/s 2022-03-17T19:04:12.1765831Z Receiving objects: 42% (324699/768219), 269.21 MiB | 18.45 MiB/s 2022-03-17T19:04:12.5654414Z Receiving objects: 43% (330335/768219), 269.21 MiB | 18.45 MiB/s 2022-03-17T19:04:12.8720058Z Receiving objects: 44% (338017/768219), 276.09 MiB | 17.85 MiB/s 2022-03-17T19:04:13.2238006Z Receiving objects: 44% (341010/768219), 283.71 MiB | 17.38 MiB/s 2022-03-17T19:04:13.7164458Z Receiving objects: 45% (345699/768219), 283.71 MiB | 17.38 MiB/s 2022-03-17T19:04:13.8704958Z Receiving objects: 46% (353381/768219), 290.87 MiB | 17.17 MiB/s 2022-03-17T19:04:14.1237397Z Receiving objects: 46% (357574/768219), 298.15 MiB | 17.07 MiB/s 2022-03-17T19:04:14.5340050Z Receiving objects: 47% (361063/768219), 298.15 MiB | 17.07 MiB/s 2022-03-17T19:04:14.8736744Z Receiving objects: 48% (368746/768219), 304.52 MiB | 16.09 MiB/s 2022-03-17T19:04:14.9001113Z Receiving objects: 48% (376105/768219), 310.92 MiB | 15.08 MiB/s 2022-03-17T19:04:15.2945435Z Receiving objects: 49% (376428/768219), 310.92 MiB | 15.08 MiB/s 2022-03-17T19:04:15.5488459Z Receiving objects: 50% (384110/768219), 317.45 MiB | 14.25 MiB/s 2022-03-17T19:04:15.7842760Z Receiving objects: 51% (391792/768219), 317.45 MiB | 14.25 MiB/s 2022-03-17T19:04:16.2732255Z Receiving objects: 52% (399474/768219), 324.51 MiB | 14.03 MiB/s 2022-03-17T19:04:17.1642589Z Receiving objects: 52% (400146/768219), 331.84 MiB | 13.90 MiB/s 2022-03-17T19:04:18.1864192Z Receiving objects: 52% (400147/768219), 339.10 MiB | 13.98 MiB/s 2022-03-17T19:04:19.2869724Z Receiving objects: 52% (400149/768219), 354.59 MiB | 14.12 MiB/s 2022-03-17T19:04:19.7348326Z Receiving objects: 52% (406248/768219), 377.96 MiB | 14.86 MiB/s 2022-03-17T19:04:19.8759627Z Receiving objects: 53% (407157/768219), 377.96 MiB | 14.86 MiB/s 2022-03-17T19:04:21.3161300Z Receiving objects: 53% (410332/768219), 385.88 MiB | 15.13 MiB/s 2022-03-17T19:04:22.3250357Z Receiving objects: 53% (410643/768219), 410.07 MiB | 15.63 MiB/s 2022-03-17T19:04:22.8390854Z Receiving objects: 53% (410643/768219), 424.96 MiB | 15.52 MiB/s 2022-03-17T19:04:22.8712276Z Receiving objects: 54% (414839/768219), 432.79 MiB | 15.50 MiB/s 2022-03-17T19:04:23.1365682Z Receiving objects: 54% (415643/768219), 432.79 MiB | 15.50 MiB/s 2022-03-17T19:04:23.4521814Z Receiving objects: 55% (422521/768219), 432.79 MiB | 15.50 MiB/s 2022-03-17T19:04:23.6895635Z Receiving objects: 56% (430203/768219), 440.96 MiB | 15.63 MiB/s 2022-03-17T19:04:24.3290601Z Receiving objects: 57% (437885/768219), 440.96 MiB | 15.63 MiB/s 2022-03-17T19:04:25.3320251Z Receiving objects: 57% (442880/768219), 456.41 MiB | 15.58 MiB/s 2022-03-17T19:04:26.3423890Z Receiving objects: 57% (442880/768219), 472.21 MiB | 15.56 MiB/s 2022-03-17T19:04:27.3436646Z Receiving objects: 57% (442880/768219), 487.09 MiB | 15.39 MiB/s 2022-03-17T19:04:28.2645129Z Receiving objects: 57% (442880/768219), 503.19 MiB | 15.59 MiB/s 2022-03-17T19:04:29.0025366Z Receiving objects: 57% (442881/768219), 511.11 MiB | 15.54 MiB/s 2022-03-17T19:04:29.8722809Z Receiving objects: 57% (442882/768219), 527.42 MiB | 15.74 MiB/s 2022-03-17T19:04:29.8853761Z Receiving objects: 57% (445371/768219), 542.68 MiB | 15.62 MiB/s 2022-03-17T19:04:30.1158059Z Receiving objects: 58% (445568/768219), 542.68 MiB | 15.62 MiB/s 2022-03-17T19:04:30.2404856Z Receiving objects: 59% (453250/768219), 542.68 MiB | 15.62 MiB/s 2022-03-17T19:04:30.4580470Z Receiving objects: 60% (460932/768219), 542.68 MiB | 15.62 MiB/s 2022-03-17T19:04:31.0383250Z Receiving objects: 61% (468614/768219), 551.07 MiB | 16.05 MiB/s 2022-03-17T19:04:31.1194494Z Receiving objects: 61% (472444/768219), 559.76 MiB | 16.14 MiB/s 2022-03-17T19:04:31.2956061Z Receiving objects: 62% (476296/768219), 559.76 MiB | 16.14 MiB/s 2022-03-17T19:04:31.3824983Z Receiving objects: 63% (483978/768219), 559.76 MiB | 16.14 MiB/s 2022-03-17T19:04:31.5241643Z Receiving objects: 64% (491661/768219), 568.47 MiB | 16.28 MiB/s 2022-03-17T19:04:31.6399141Z Receiving objects: 65% (499343/768219), 568.47 MiB | 16.28 MiB/s 2022-03-17T19:04:31.8312350Z Receiving objects: 66% (507025/768219), 568.47 MiB | 16.28 MiB/s 2022-03-17T19:04:31.8748127Z Receiving objects: 67% (514707/768219), 568.47 MiB | 16.28 MiB/s 2022-03-17T19:04:31.9517074Z Receiving objects: 67% (517455/768219), 577.21 MiB | 16.44 MiB/s 2022-03-17T19:04:32.0662085Z Receiving objects: 68% (522389/768219), 577.21 MiB | 16.44 MiB/s 2022-03-17T19:04:32.2496270Z Receiving objects: 69% (530072/768219), 577.21 MiB | 16.44 MiB/s 2022-03-17T19:04:32.3580993Z Receiving objects: 70% (537754/768219), 577.21 MiB | 16.44 MiB/s 2022-03-17T19:04:32.8894734Z Receiving objects: 71% (545436/768219), 585.60 MiB | 16.54 MiB/s 2022-03-17T19:04:32.8937510Z Receiving objects: 71% (552830/768219), 592.65 MiB | 16.16 MiB/s 2022-03-17T19:04:33.0594341Z Receiving objects: 72% (553118/768219), 592.65 MiB | 16.16 MiB/s 2022-03-17T19:04:33.2666530Z Receiving objects: 73% (560800/768219), 592.65 MiB | 16.16 MiB/s 2022-03-17T19:04:33.4688792Z Receiving objects: 74% (568483/768219), 592.65 MiB | 16.16 MiB/s 2022-03-17T19:04:33.6776085Z Receiving objects: 75% (576165/768219), 601.37 MiB | 16.12 MiB/s 2022-03-17T19:04:33.8120934Z Receiving objects: 76% (583847/768219), 601.37 MiB | 16.12 MiB/s 2022-03-17T19:04:33.8812651Z Receiving objects: 77% (591529/768219), 601.37 MiB | 16.12 MiB/s 2022-03-17T19:04:34.0120040Z Receiving objects: 77% (592426/768219), 601.37 MiB | 16.12 MiB/s 2022-03-17T19:04:34.3256751Z Receiving objects: 78% (599211/768219), 608.92 MiB | 15.91 MiB/s 2022-03-17T19:04:34.6360758Z Receiving objects: 79% (606894/768219), 608.92 MiB | 15.91 MiB/s 2022-03-17T19:04:34.8305367Z Receiving objects: 80% (614576/768219), 615.88 MiB | 15.68 MiB/s 2022-03-17T19:04:34.9198559Z Receiving objects: 81% (622258/768219), 615.88 MiB | 15.68 MiB/s 2022-03-17T19:04:35.1623249Z Receiving objects: 81% (623090/768219), 615.88 MiB | 15.68 MiB/s 2022-03-17T19:04:35.3713127Z Receiving objects: 82% (629940/768219), 621.58 MiB | 15.09 MiB/s 2022-03-17T19:04:35.6939504Z Receiving objects: 83% (637622/768219), 621.58 MiB | 15.09 MiB/s 2022-03-17T19:04:35.8834955Z Receiving objects: 84% (645304/768219), 627.45 MiB | 14.47 MiB/s 2022-03-17T19:04:37.0396635Z Receiving objects: 84% (648406/768219), 627.45 MiB | 14.47 MiB/s 2022-03-17T19:04:38.0396954Z Receiving objects: 84% (649939/768219), 643.77 MiB | 12.41 MiB/s 2022-03-17T19:04:39.0400145Z Receiving objects: 84% (649939/768219), 653.71 MiB | 11.37 MiB/s 2022-03-17T19:04:40.0875600Z Receiving objects: 84% (649939/768219), 663.26 MiB | 10.48 MiB/s 2022-03-17T19:04:40.8288472Z Receiving objects: 84% (649939/768219), 672.07 MiB | 9.79 MiB/s 2022-03-17T19:04:40.8704406Z Receiving objects: 85% (652987/768219), 676.63 MiB | 9.54 MiB/s 2022-03-17T19:04:41.2959107Z Receiving objects: 85% (654029/768219), 676.63 MiB | 9.54 MiB/s 2022-03-17T19:04:41.8708331Z Receiving objects: 86% (660669/768219), 680.67 MiB | 9.26 MiB/s 2022-03-17T19:04:41.8718431Z Receiving objects: 86% (668320/768219), 685.06 MiB | 9.07 MiB/s 2022-03-17T19:04:42.8758054Z Receiving objects: 87% (668351/768219), 685.06 MiB | 9.07 MiB/s 2022-03-17T19:04:42.9271182Z Receiving objects: 87% (675802/768219), 692.57 MiB | 8.54 MiB/s 2022-03-17T19:04:43.8295313Z Receiving objects: 88% (676033/768219), 692.57 MiB | 8.54 MiB/s 2022-03-17T19:04:43.8829428Z Receiving objects: 89% (683715/768219), 699.34 MiB | 7.90 MiB/s 2022-03-17T19:04:44.7211661Z Receiving objects: 89% (684390/768219), 699.34 MiB | 7.90 MiB/s 2022-03-17T19:04:44.8740919Z Receiving objects: 90% (691398/768219), 706.25 MiB | 7.56 MiB/s 2022-03-17T19:04:45.4489578Z Receiving objects: 90% (692974/768219), 706.25 MiB | 7.56 MiB/s 2022-03-17T19:04:45.8706350Z Receiving objects: 91% (699080/768219), 709.54 MiB | 7.28 MiB/s 2022-03-17T19:04:46.1322421Z Receiving objects: 91% (703821/768219), 712.89 MiB | 7.13 MiB/s 2022-03-17T19:04:46.5306509Z Receiving objects: 92% (706762/768219), 716.43 MiB | 6.94 MiB/s 2022-03-17T19:04:46.8549977Z Receiving objects: 93% (714444/768219), 716.43 MiB | 6.94 MiB/s 2022-03-17T19:04:46.8704056Z Receiving objects: 94% (722126/768219), 720.22 MiB | 6.85 MiB/s 2022-03-17T19:04:47.1553080Z Receiving objects: 94% (722800/768219), 720.22 MiB | 6.85 MiB/s 2022-03-17T19:04:47.6625639Z Receiving objects: 95% (729809/768219), 724.82 MiB | 7.13 MiB/s 2022-03-17T19:04:47.8710664Z Receiving objects: 96% (737491/768219), 729.11 MiB | 7.42 MiB/s 2022-03-17T19:04:47.9416946Z Receiving objects: 96% (743077/768219), 729.11 MiB | 7.42 MiB/s 2022-03-17T19:04:48.1903063Z Receiving objects: 97% (745173/768219), 729.11 MiB | 7.42 MiB/s 2022-03-17T19:04:48.5445583Z Receiving objects: 98% (752855/768219), 733.38 MiB | 7.55 MiB/s 2022-03-17T19:04:48.8703723Z Receiving objects: 99% (760537/768219), 733.38 MiB | 7.55 MiB/s 2022-03-17T19:04:48.9761231Z Receiving objects: 99% (765842/768219), 737.67 MiB | 7.71 MiB/s 2022-03-17T19:04:48.9763204Z remote: Total 768219 (delta 245), reused 199 (delta 103), pack-reused 767821 2022-03-17T19:04:48.9784116Z Receiving objects: 100% (768219/768219), 737.67 MiB | 7.71 MiB/s 2022-03-17T19:04:48.9785113Z Receiving objects: 100% (768219/768219), 740.56 MiB | 11.20 MiB/s, done. 2022-03-17T19:04:49.1141151Z Resolving deltas: 0% (0/622271) 2022-03-17T19:04:49.1462111Z Resolving deltas: 1% (6223/622271) 2022-03-17T19:04:49.1803241Z Resolving deltas: 2% (12446/622271) 2022-03-17T19:04:49.2091354Z Resolving deltas: 3% (18669/622271) 2022-03-17T19:04:49.2877812Z Resolving deltas: 4% (24892/622271) 2022-03-17T19:04:49.3211007Z Resolving deltas: 5% (31121/622271) 2022-03-17T19:04:49.3918698Z Resolving deltas: 6% (37337/622271) 2022-03-17T19:04:49.5565836Z Resolving deltas: 7% (43559/622271) 2022-03-17T19:04:49.5901573Z Resolving deltas: 8% (49782/622271) 2022-03-17T19:04:49.6253959Z Resolving deltas: 9% (56005/622271) 2022-03-17T19:04:49.6536899Z Resolving deltas: 10% (62228/622271) 2022-03-17T19:04:49.6876045Z Resolving deltas: 11% (68450/622271) 2022-03-17T19:04:49.7168235Z Resolving deltas: 12% (74673/622271) 2022-03-17T19:04:49.7601709Z Resolving deltas: 13% (80897/622271) 2022-03-17T19:04:50.0869797Z Resolving deltas: 14% (87118/622271) 2022-03-17T19:04:50.2713391Z Resolving deltas: 14% (91704/622271) 2022-03-17T19:04:50.4629198Z Resolving deltas: 15% (93341/622271) 2022-03-17T19:04:50.5406045Z Resolving deltas: 16% (99564/622271) 2022-03-17T19:04:50.5794915Z Resolving deltas: 17% (105788/622271) 2022-03-17T19:04:50.6245676Z Resolving deltas: 18% (112010/622271) 2022-03-17T19:04:50.6710644Z Resolving deltas: 19% (118232/622271) 2022-03-17T19:04:50.7240498Z Resolving deltas: 20% (124457/622271) 2022-03-17T19:04:50.8012978Z Resolving deltas: 21% (130678/622271) 2022-03-17T19:04:50.8292032Z Resolving deltas: 22% (136900/622271) 2022-03-17T19:04:50.8516919Z Resolving deltas: 23% (143124/622271) 2022-03-17T19:04:50.8902655Z Resolving deltas: 24% (149346/622271) 2022-03-17T19:04:50.9236407Z Resolving deltas: 25% (155568/622271) 2022-03-17T19:04:50.9992715Z Resolving deltas: 26% (161791/622271) 2022-03-17T19:04:51.0222468Z Resolving deltas: 27% (168015/622271) 2022-03-17T19:04:51.0868260Z Resolving deltas: 28% (174239/622271) 2022-03-17T19:04:51.0996732Z Resolving deltas: 28% (178195/622271) 2022-03-17T19:04:51.1187579Z Resolving deltas: 29% (180460/622271) 2022-03-17T19:04:51.1471108Z Resolving deltas: 30% (186682/622271) 2022-03-17T19:04:51.2689544Z Resolving deltas: 31% (192905/622271) 2022-03-17T19:04:51.2879079Z Resolving deltas: 32% (199128/622271) 2022-03-17T19:04:51.3071017Z Resolving deltas: 33% (205351/622271) 2022-03-17T19:04:51.3292580Z Resolving deltas: 34% (211574/622271) 2022-03-17T19:04:51.3491837Z Resolving deltas: 35% (217795/622271) 2022-03-17T19:04:51.3725857Z Resolving deltas: 36% (224020/622271) 2022-03-17T19:04:51.3925100Z Resolving deltas: 37% (230242/622271) 2022-03-17T19:04:51.4174725Z Resolving deltas: 38% (236465/622271) 2022-03-17T19:04:51.4403217Z Resolving deltas: 39% (242688/622271) 2022-03-17T19:04:51.4641593Z Resolving deltas: 40% (248909/622271) 2022-03-17T19:04:51.4963263Z Resolving deltas: 41% (255132/622271) 2022-03-17T19:04:51.5259215Z Resolving deltas: 42% (261354/622271) 2022-03-17T19:04:51.5466999Z Resolving deltas: 43% (267577/622271) 2022-03-17T19:04:51.5728547Z Resolving deltas: 44% (273800/622271) 2022-03-17T19:04:51.5990050Z Resolving deltas: 45% (280022/622271) 2022-03-17T19:04:51.6238423Z Resolving deltas: 46% (286247/622271) 2022-03-17T19:04:51.6659317Z Resolving deltas: 47% (292468/622271) 2022-03-17T19:04:51.6931862Z Resolving deltas: 48% (298691/622271) 2022-03-17T19:04:51.7215092Z Resolving deltas: 49% (304913/622271) 2022-03-17T19:04:51.7448012Z Resolving deltas: 50% (311136/622271) 2022-03-17T19:04:51.7680626Z Resolving deltas: 51% (317359/622271) 2022-03-17T19:04:51.7940445Z Resolving deltas: 52% (323582/622271) 2022-03-17T19:04:51.8742794Z Resolving deltas: 53% (329808/622271) 2022-03-17T19:04:51.9091150Z Resolving deltas: 54% (336027/622271) 2022-03-17T19:04:51.9306193Z Resolving deltas: 55% (342251/622271) 2022-03-17T19:04:51.9520854Z Resolving deltas: 56% (348472/622271) 2022-03-17T19:04:51.9787692Z Resolving deltas: 57% (354701/622271) 2022-03-17T19:04:52.0087043Z Resolving deltas: 58% (360918/622271) 2022-03-17T19:04:52.0286401Z Resolving deltas: 59% (367140/622271) 2022-03-17T19:04:52.0502385Z Resolving deltas: 60% (373363/622271) 2022-03-17T19:04:52.0847999Z Resolving deltas: 61% (379589/622271) 2022-03-17T19:04:52.0865686Z Resolving deltas: 62% (385809/622271) 2022-03-17T19:04:52.2301529Z Resolving deltas: 62% (386390/622271) 2022-03-17T19:04:52.2512515Z Resolving deltas: 63% (392032/622271) 2022-03-17T19:04:52.2746915Z Resolving deltas: 64% (398256/622271) 2022-03-17T19:04:52.2956990Z Resolving deltas: 65% (404478/622271) 2022-03-17T19:04:52.3262852Z Resolving deltas: 66% (410700/622271) 2022-03-17T19:04:52.3574846Z Resolving deltas: 67% (416926/622271) 2022-03-17T19:04:52.3783240Z Resolving deltas: 68% (423148/622271) 2022-03-17T19:04:52.4072131Z Resolving deltas: 69% (429367/622271) 2022-03-17T19:04:52.4290204Z Resolving deltas: 70% (435590/622271) 2022-03-17T19:04:52.4494299Z Resolving deltas: 71% (441815/622271) 2022-03-17T19:04:52.5056049Z Resolving deltas: 72% (448038/622271) 2022-03-17T19:04:52.5600558Z Resolving deltas: 73% (454258/622271) 2022-03-17T19:04:52.5861810Z Resolving deltas: 74% (460482/622271) 2022-03-17T19:04:52.7087046Z Resolving deltas: 75% (466704/622271) 2022-03-17T19:04:52.7831371Z Resolving deltas: 76% (472926/622271) 2022-03-17T19:04:52.8098400Z Resolving deltas: 77% (479149/622271) 2022-03-17T19:04:52.8343687Z Resolving deltas: 78% (485372/622271) 2022-03-17T19:04:52.8591343Z Resolving deltas: 79% (491597/622271) 2022-03-17T19:04:52.8915464Z Resolving deltas: 80% (497817/622271) 2022-03-17T19:04:52.9186603Z Resolving deltas: 81% (504040/622271) 2022-03-17T19:04:52.9587824Z Resolving deltas: 82% (510263/622271) 2022-03-17T19:04:52.9847923Z Resolving deltas: 83% (516485/622271) 2022-03-17T19:04:53.0091608Z Resolving deltas: 84% (522708/622271) 2022-03-17T19:04:53.0325806Z Resolving deltas: 85% (528931/622271) 2022-03-17T19:04:53.0627726Z Resolving deltas: 86% (535155/622271) 2022-03-17T19:04:53.0865951Z Resolving deltas: 87% (541376/622271) 2022-03-17T19:04:53.0979218Z Resolving deltas: 87% (545993/622271) 2022-03-17T19:04:53.1308029Z Resolving deltas: 88% (547599/622271) 2022-03-17T19:04:53.1655784Z Resolving deltas: 89% (553822/622271) 2022-03-17T19:04:53.1990236Z Resolving deltas: 90% (560044/622271) 2022-03-17T19:04:53.2283752Z Resolving deltas: 91% (566267/622271) 2022-03-17T19:04:53.2564381Z Resolving deltas: 92% (572490/622271) 2022-03-17T19:04:53.2821838Z Resolving deltas: 93% (578713/622271) 2022-03-17T19:04:53.3060544Z Resolving deltas: 94% (584935/622271) 2022-03-17T19:04:53.3279054Z Resolving deltas: 95% (591158/622271) 2022-03-17T19:04:53.3495191Z Resolving deltas: 96% (597382/622271) 2022-03-17T19:04:53.3728984Z Resolving deltas: 97% (603603/622271) 2022-03-17T19:04:53.3945354Z Resolving deltas: 98% (609828/622271) 2022-03-17T19:04:53.4184957Z Resolving deltas: 99% (616049/622271) 2022-03-17T19:04:53.4186282Z Resolving deltas: 100% (622271/622271) 2022-03-17T19:04:53.4186635Z Resolving deltas: 100% (622271/622271), done. 2022-03-17T19:04:57.9059027Z From https://github.com/pytorch/pytorch 2022-03-17T19:04:57.9060918Z * [new branch] 1.6 -> origin/1.6 2022-03-17T19:04:57.9062194Z * [new branch] 1.6_doc_references -> origin/1.6_doc_references 2022-03-17T19:04:57.9065423Z * [new branch] 1.7 -> origin/1.7 2022-03-17T19:04:57.9069287Z * [new branch] 20_12_3_devel -> origin/20_12_3_devel 2022-03-17T19:04:57.9073005Z * [new branch] 21250-2 -> origin/21250-2 2022-03-17T19:04:57.9076925Z * [new branch] 32495checkout -> origin/32495checkout 2022-03-17T19:04:57.9081164Z * [new branch] 42835-cherry-pick -> origin/42835-cherry-pick 2022-03-17T19:04:57.9085679Z * [new branch] 53867-memory-planninger-output-tensor -> origin/53867-memory-planninger-output-tensor 2022-03-17T19:04:57.9089066Z * [new branch] 71806_migrate_default_branch_to_main -> origin/71806_migrate_default_branch_to_main 2022-03-17T19:04:57.9092680Z * [new branch] 71806_migrate_master_branch_to_main -> origin/71806_migrate_master_branch_to_main 2022-03-17T19:04:57.9096547Z * [new branch] 71806_support_main_branch_in_github_workflows -> origin/71806_support_main_branch_in_github_workflows 2022-03-17T19:04:57.9099940Z * [new branch] Add-new-process-group-_internal_ucc -> origin/Add-new-process-group-_internal_ucc 2022-03-17T19:04:57.9103163Z * [new branch] ArchiveStream -> origin/ArchiveStream 2022-03-17T19:04:57.9107127Z * [new branch] BUILD_SPLIT_CUDA_AUTO_DETECT_CPP_EXT -> origin/BUILD_SPLIT_CUDA_AUTO_DETECT_CPP_EXT 2022-03-17T19:04:57.9110388Z * [new branch] CUDA_TEST -> origin/CUDA_TEST 2022-03-17T19:04:57.9114127Z * [new branch] Chillee-patch-1 -> origin/Chillee-patch-1 2022-03-17T19:04:57.9117541Z * [new branch] CodeGenRefactor -> origin/CodeGenRefactor 2022-03-17T19:04:57.9121113Z * [new branch] CodeGenRefactor_diff -> origin/CodeGenRefactor_diff 2022-03-17T19:04:57.9125066Z * [new branch] DIFF_B_1 -> origin/DIFF_B_1 2022-03-17T19:04:57.9128512Z * [new branch] DIFF_B_2 -> origin/DIFF_B_2 2022-03-17T19:04:57.9132249Z * [new branch] DataPipeForkDemux -> origin/DataPipeForkDemux 2022-03-17T19:04:57.9133582Z * [new branch] DataPipesAPI -> origin/DataPipesAPI 2022-03-17T19:04:57.9135390Z * [new branch] DataPipesAPIClarity -> origin/DataPipesAPIClarity 2022-03-17T19:04:57.9136623Z * [new branch] DataPipes_Simple_Unbatch -> origin/DataPipes_Simple_Unbatch 2022-03-17T19:04:57.9137862Z * [new branch] FE2774773A77F1D7EA51AFAD32EA53C9 -> origin/FE2774773A77F1D7EA51AFAD32EA53C9 2022-03-17T19:04:57.9139749Z * [new branch] FOO1 -> origin/FOO1 2022-03-17T19:04:57.9141105Z * [new branch] Fix-master-due-to-62520 -> origin/Fix-master-due-to-62520 2022-03-17T19:04:57.9142579Z * [new branch] FixCombiningWarning -> origin/FixCombiningWarning 2022-03-17T19:04:57.9143998Z * [new branch] Improve_SetRNGState_Doc -> origin/Improve_SetRNGState_Doc 2022-03-17T19:04:57.9145421Z * [new branch] IvanYashchuk-patch-1 -> origin/IvanYashchuk-patch-1 2022-03-17T19:04:57.9146888Z * [new branch] IvanYashchuk-patch-2 -> origin/IvanYashchuk-patch-2 2022-03-17T19:04:57.9148181Z * [new branch] MTH_DDP -> origin/MTH_DDP 2022-03-17T19:04:57.9149474Z * [new branch] RemoveTensorOptionsRC -> origin/RemoveTensorOptionsRC 2022-03-17T19:04:57.9150938Z * [new branch] RemoveTensorOptionsRC_2 -> origin/RemoveTensorOptionsRC_2 2022-03-17T19:04:57.9152291Z * [new branch] SsnL-patch-1 -> origin/SsnL-patch-1 2022-03-17T19:04:57.9153551Z * [new branch] SsnL-patch-12 -> origin/SsnL-patch-12 2022-03-17T19:04:57.9155119Z * [new branch] SsnL-patch-4 -> origin/SsnL-patch-4 2022-03-17T19:04:57.9156413Z * [new branch] SsnL-patch-8 -> origin/SsnL-patch-8 2022-03-17T19:04:57.9157651Z * [new branch] T103677508-maxren -> origin/T103677508-maxren 2022-03-17T19:04:57.9158818Z * [new branch] T105015700 -> origin/T105015700 2022-03-17T19:04:57.9159937Z * [new branch] T105015700-2 -> origin/T105015700-2 2022-03-17T19:04:57.9161071Z * [new branch] T105015700-3 -> origin/T105015700-3 2022-03-17T19:04:57.9163291Z * [new branch] T98160523 -> origin/T98160523 2022-03-17T19:04:57.9164419Z * [new branch] T98160523_2 -> origin/T98160523_2 2022-03-17T19:04:57.9165596Z * [new branch] TO_WORK_BRANCH -> origin/TO_WORK_BRANCH 2022-03-17T19:04:57.9166884Z * [new branch] TarFileReaderFix -> origin/TarFileReaderFix 2022-03-17T19:04:57.9168506Z * [new branch] TorchDataDocsUpdate -> origin/TorchDataDocsUpdate 2022-03-17T19:04:57.9169931Z * [new branch] Update-libtorch-path -> origin/Update-libtorch-path 2022-03-17T19:04:57.9171424Z * [new branch] Update-libtorch-path-2 -> origin/Update-libtorch-path-2 2022-03-17T19:04:57.9172881Z * [new branch] Update-test-distribute -> origin/Update-test-distribute 2022-03-17T19:04:57.9174100Z * [new branch] Update-test-rpc -> origin/Update-test-rpc 2022-03-17T19:04:57.9175021Z * [new branch] Update-test-vulkan -> origin/Update-test-vulkan 2022-03-17T19:04:57.9175991Z * [new branch] VitalyFedyunin-patch-1 -> origin/VitalyFedyunin-patch-1 2022-03-17T19:04:57.9178102Z * [new branch] __gg_suo/1603135434 -> origin/__gg_suo/1603135434 2022-03-17T19:04:57.9179207Z * [new branch] __gg_suo/1645154242 -> origin/__gg_suo/1645154242 2022-03-17T19:04:57.9180428Z * [new branch] __gg_suo/1647471426 -> origin/__gg_suo/1647471426 2022-03-17T19:04:57.9181715Z * [new branch] _aminmax_deprecation -> origin/_aminmax_deprecation 2022-03-17T19:04:57.9183188Z * [new branch] activation_functions_i_opinfos -> origin/activation_functions_i_opinfos 2022-03-17T19:04:57.9184473Z * [new branch] adapative_autorange_redo -> origin/adapative_autorange_redo 2022-03-17T19:04:57.9185902Z * [new branch] add-docs-rule-GH1 -> origin/add-docs-rule-GH1 2022-03-17T19:04:57.9187388Z * [new branch] add-print-test-stats-step-to-macos -> origin/add-print-test-stats-step-to-macos 2022-03-17T19:04:57.9188561Z * [new branch] add-test-whl -> origin/add-test-whl 2022-03-17T19:04:57.9189879Z * [new branch] add-wheel-test-job -> origin/add-wheel-test-job 2022-03-17T19:04:57.9191453Z * [new branch] add-wheel-test-job-2 -> origin/add-wheel-test-job-2 2022-03-17T19:04:57.9192677Z * [new branch] add_code_filter_for_TS -> origin/add_code_filter_for_TS 2022-03-17T19:04:57.9194024Z * [new branch] add_default_args -> origin/add_default_args 2022-03-17T19:04:57.9195399Z * [new branch] add_llvm -> origin/add_llvm 2022-03-17T19:04:57.9196876Z * [new branch] add_monkeytype_to_tests -> origin/add_monkeytype_to_tests 2022-03-17T19:04:57.9198361Z * [new branch] add_multiline_support_for_typing_variables -> origin/add_multiline_support_for_typing_variables 2022-03-17T19:04:57.9199737Z * [new branch] add_support_for_optional_tensor -> origin/add_support_for_optional_tensor 2022-03-17T19:04:57.9200998Z * [new branch] add_tags_to_Parameter_dev1 -> origin/add_tags_to_Parameter_dev1 2022-03-17T19:04:57.9202750Z * [new branch] add_tests_for_out -> origin/add_tests_for_out 2022-03-17T19:04:57.9203928Z * [new branch] addcmul_opinfo -> origin/addcmul_opinfo 2022-03-17T19:04:57.9205195Z * [new branch] adjoint -> origin/adjoint 2022-03-17T19:04:57.9207083Z * [new branch] alanwaketan/ddp_fallback -> origin/alanwaketan/ddp_fallback 2022-03-17T19:04:57.9208456Z * [new branch] alanwaketan/grep_opt_params -> origin/alanwaketan/grep_opt_params 2022-03-17T19:04:57.9209719Z * [new branch] alanwaketan/lazy_bench -> origin/alanwaketan/lazy_bench 2022-03-17T19:04:57.9211148Z * [new branch] alanwaketan/mark_step -> origin/alanwaketan/mark_step 2022-03-17T19:04:57.9212302Z * [new branch] alanwaketan/tb -> origin/alanwaketan/tb 2022-03-17T19:04:57.9213787Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2022-03-17T19:04:57.9215160Z * [new branch] albanD-patch-2 -> origin/albanD-patch-2 2022-03-17T19:04:57.9216637Z * [new branch] albanD-patch-3 -> origin/albanD-patch-3 2022-03-17T19:04:57.9218612Z * [new branch] albanD/codeowner-binding -> origin/albanD/codeowner-binding 2022-03-17T19:04:57.9219720Z * [new branch] albanD/contrib_update -> origin/albanD/contrib_update 2022-03-17T19:04:57.9221336Z * [new branch] alband/fix_gil_bug_distributed -> origin/alband/fix_gil_bug_distributed 2022-03-17T19:04:57.9222746Z * [new branch] aliasanalysis -> origin/aliasanalysis 2022-03-17T19:04:57.9223987Z * [new branch] aliases_in_python -> origin/aliases_in_python 2022-03-17T19:04:57.9225331Z * [new branch] all_codegen_ops -> origin/all_codegen_ops 2022-03-17T19:04:57.9226595Z * [new branch] allgathercoalesced -> origin/allgathercoalesced 2022-03-17T19:04:57.9227987Z * [new branch] always_wrap -> origin/always_wrap 2022-03-17T19:04:57.9229168Z * [new branch] andrew-temp -> origin/andrew-temp 2022-03-17T19:04:57.9230613Z * [new branch] android_fbgemm_off -> origin/android_fbgemm_off 2022-03-17T19:04:57.9231879Z * [new branch] android_fbgemm_off_pr -> origin/android_fbgemm_off_pr 2022-03-17T19:04:57.9233328Z * [new branch] android_imageutils_chlast -> origin/android_imageutils_chlast 2022-03-17T19:04:57.9234570Z * [new branch] android_memory_format -> origin/android_memory_format 2022-03-17T19:04:57.9236017Z * [new branch] annotate_parameter_as_tensor -> origin/annotate_parameter_as_tensor 2022-03-17T19:04:57.9237404Z * [new branch] aot-compiler-shape-quant -> origin/aot-compiler-shape-quant 2022-03-17T19:04:57.9238764Z * [new branch] api_release -> origin/api_release 2022-03-17T19:04:57.9240050Z * [new branch] applySelect_exp -> origin/applySelect_exp 2022-03-17T19:04:57.9241783Z * [new branch] as_tensor_docs -> origin/as_tensor_docs 2022-03-17T19:04:57.9243270Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2022-03-17T19:04:57.9244648Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2022-03-17T19:04:57.9245968Z * [new branch] augment_inline_callstack -> origin/augment_inline_callstack 2022-03-17T19:04:57.9247968Z * [new branch] axitkhurana/docs-typo-fix -> origin/axitkhurana/docs-typo-fix 2022-03-17T19:04:57.9249324Z * [new branch] backport_api -> origin/backport_api 2022-03-17T19:04:57.9250719Z * [new branch] backward -> origin/backward 2022-03-17T19:04:57.9252557Z * [new branch] bahuang/test -> origin/bahuang/test 2022-03-17T19:04:57.9253834Z * [new branch] bahuang/test2 -> origin/bahuang/test2 2022-03-17T19:04:57.9255822Z * [new branch] base/1.5 -> origin/base/1.5 2022-03-17T19:04:57.9256941Z * [new branch] base/6f8328e -> origin/base/6f8328e 2022-03-17T19:04:57.9258256Z * [new branch] base/9e6877c -> origin/base/9e6877c 2022-03-17T19:04:57.9259980Z * [new branch] base/b7b481b -> origin/base/b7b481b 2022-03-17T19:04:57.9261386Z * [new branch] batch_norm_folding_traced_modules -> origin/batch_norm_folding_traced_modules 2022-03-17T19:04:57.9262910Z * [new branch] bc_for_torchvision_nonlinearity_fanmode -> origin/bc_for_torchvision_nonlinearity_fanmode 2022-03-17T19:04:57.9263987Z * [new branch] bench_fix -> origin/bench_fix 2022-03-17T19:04:57.9265636Z * [new branch] bert_refactor -> origin/bert_refactor 2022-03-17T19:04:57.9267161Z * [new branch] bertmaher/nncops -> origin/bertmaher/nncops 2022-03-17T19:04:57.9268504Z * [new branch] bf16 -> origin/bf16 2022-03-17T19:04:57.9269870Z * [new branch] bfgsfix -> origin/bfgsfix 2022-03-17T19:04:57.9271695Z * [new branch] bfloat-elementwise2 -> origin/bfloat-elementwise2 2022-03-17T19:04:57.9272943Z * [new branch] binbao/batch_norm -> origin/binbao/batch_norm 2022-03-17T19:04:57.9274260Z * [new branch] binbao/fast_tracing -> origin/binbao/fast_tracing 2022-03-17T19:04:57.9275501Z * [new branch] binbao/index_tensor -> origin/binbao/index_tensor 2022-03-17T19:04:57.9276872Z * [new branch] binbao/measure_tracing -> origin/binbao/measure_tracing 2022-03-17T19:04:57.9278064Z * [new branch] binbao/merge -> origin/binbao/merge 2022-03-17T19:04:57.9279354Z * [new branch] binbao/merge2 -> origin/binbao/merge2 2022-03-17T19:04:57.9280641Z * [new branch] binbao/merge3 -> origin/binbao/merge3 2022-03-17T19:04:57.9282238Z * [new branch] bincount_opinfo -> origin/bincount_opinfo 2022-03-17T19:04:57.9283641Z * [new branch] boo2 -> origin/boo2 2022-03-17T19:04:57.9285027Z * [new branch] bowang_test -> origin/bowang_test 2022-03-17T19:04:57.9286923Z * [new branch] bowbao/extend_onnx_rule_files -> origin/bowbao/extend_onnx_rule_files 2022-03-17T19:04:57.9288073Z * [new branch] bowbao/fix_lstm_regression -> origin/bowbao/fix_lstm_regression 2022-03-17T19:04:57.9289278Z * [new branch] bowbao/onnx_bart -> origin/bowbao/onnx_bart 2022-03-17T19:04:57.9290692Z * [new branch] bowbao/onnx_embedding_renorm -> origin/bowbao/onnx_embedding_renorm 2022-03-17T19:04:57.9291960Z * [new branch] bowbao/onnx_infer_scope_func -> origin/bowbao/onnx_infer_scope_func 2022-03-17T19:04:57.9293378Z * [new branch] bowbao/onnx_local_function_attrs -> origin/bowbao/onnx_local_function_attrs 2022-03-17T19:04:57.9294553Z * [new branch] bowbao/onnx_tracing_len -> origin/bowbao/onnx_tracing_len 2022-03-17T19:04:57.9296724Z * [new branch] boyuantan/print-sccache-only-on-builds -> origin/boyuantan/print-sccache-only-on-builds 2022-03-17T19:04:57.9297737Z * [new branch] break-llvm -> origin/break-llvm 2022-03-17T19:04:57.9299082Z * [new branch] brianjo-doctest -> origin/brianjo-doctest 2022-03-17T19:04:57.9300831Z * [new branch] brianjo-expand-fix -> origin/brianjo-expand-fix 2022-03-17T19:04:57.9302136Z * [new branch] brianjo-jitv2fix -> origin/brianjo-jitv2fix 2022-03-17T19:04:57.9303559Z * [new branch] brianjo-torchrec-domainlist -> origin/brianjo-torchrec-domainlist 2022-03-17T19:04:57.9304887Z * [new branch] brianjo_versions -> origin/brianjo_versions 2022-03-17T19:04:57.9306361Z * [new branch] bucketize_opinfo -> origin/bucketize_opinfo 2022-03-17T19:04:57.9307578Z * [new branch] build-test-whl -> origin/build-test-whl 2022-03-17T19:04:57.9309024Z * [new branch] build-whl-for-testing -> origin/build-whl-for-testing 2022-03-17T19:04:57.9310351Z * [new branch] build-whl-package-2 -> origin/build-whl-package-2 2022-03-17T19:04:57.9311673Z * [new branch] build-whl-package-3 -> origin/build-whl-package-3 2022-03-17T19:04:57.9313106Z * [new branch] build-whl-package-4 -> origin/build-whl-package-4 2022-03-17T19:04:57.9314820Z * [new branch] build/1.3.0 -> origin/build/1.3.0 2022-03-17T19:04:57.9316084Z * [new branch] build/1.3.1 -> origin/build/1.3.1 2022-03-17T19:04:57.9317412Z * [new branch] cached_functions -> origin/cached_functions 2022-03-17T19:04:57.9318796Z * [new branch] cacherator -> origin/cacherator 2022-03-17T19:04:57.9320346Z * [new branch] caffe_2_master_references_to_main -> origin/caffe_2_master_references_to_main 2022-03-17T19:04:57.9321748Z * [new branch] cbalioglu-patch-1 -> origin/cbalioglu-patch-1 2022-03-17T19:04:57.9323469Z * [new branch] change_ns_cuda_mar_10 -> origin/change_ns_cuda_mar_10 2022-03-17T19:04:57.9324890Z * [new branch] check-cupti-so -> origin/check-cupti-so 2022-03-17T19:04:57.9326176Z * [new branch] checkout-pytorch -> origin/checkout-pytorch 2022-03-17T19:04:57.9327659Z * [new branch] cherry-pick-58572 -> origin/cherry-pick-58572 2022-03-17T19:04:57.9329039Z * [new branch] cherry-pick-58573 -> origin/cherry-pick-58573 2022-03-17T19:04:57.9330595Z * [new branch] cherry-pick-fix-refcounting -> origin/cherry-pick-fix-refcounting 2022-03-17T19:04:57.9331921Z * [new branch] cherrypick_xla_codegen -> origin/cherrypick_xla_codegen 2022-03-17T19:04:57.9334151Z * [new branch] chesterliu/dev/fix-warnings-for-msvc-1 -> origin/chesterliu/dev/fix-warnings-for-msvc-1 2022-03-17T19:04:57.9334932Z * [new branch] chesterliu/dev/skip-nccl-1 -> origin/chesterliu/dev/skip-nccl-1 2022-03-17T19:04:57.9336421Z * [new branch] chown -> origin/chown 2022-03-17T19:04:57.9337772Z * [new branch] ci -> origin/ci 2022-03-17T19:04:57.9339564Z * [new branch] ci-all/aar_headers_test_app_2 -> origin/ci-all/aar_headers_test_app_2 2022-03-17T19:04:57.9340798Z * [new branch] ci-all/add-back-11.1-ci -> origin/ci-all/add-back-11.1-ci 2022-03-17T19:04:57.9342303Z * [new branch] ci-all/add-initial-linux-ci-gha -> origin/ci-all/add-initial-linux-ci-gha 2022-03-17T19:04:57.9343798Z * [new branch] ci-all/agolynski/mpicuda -> origin/ci-all/agolynski/mpicuda 2022-03-17T19:04:57.9345474Z * [new branch] ci-all/albanD/fwad_opinfo -> origin/ci-all/albanD/fwad_opinfo 2022-03-17T19:04:57.9346736Z * [new branch] ci-all/android_aar_headers_ci -> origin/ci-all/android_aar_headers_ci 2022-03-17T19:04:57.9348171Z * [new branch] ci-all/awgu -> origin/ci-all/awgu 2022-03-17T19:04:57.9349442Z * [new branch] ci-all/b_06e66e29 -> origin/ci-all/b_06e66e29 2022-03-17T19:04:57.9350830Z * [new branch] ci-all/baddbmm_port -> origin/ci-all/baddbmm_port 2022-03-17T19:04:57.9352346Z * [new branch] ci-all/batchnorm3d-channels-last-cudnn-spatial-persistent -> origin/ci-all/batchnorm3d-channels-last-cudnn-spatial-persistent 2022-03-17T19:04:57.9353309Z * [new branch] ci-all/better-numerics-test -> origin/ci-all/better-numerics-test 2022-03-17T19:04:57.9354735Z * [new branch] ci-all/bfloat16-cuda-tests -> origin/ci-all/bfloat16-cuda-tests 2022-03-17T19:04:57.9356068Z * [new branch] ci-all/binary_debug_test -> origin/ci-all/binary_debug_test 2022-03-17T19:04:57.9357778Z * [new branch] ci-all/ci-all/solve-internal-rewrite -> origin/ci-all/ci-all/solve-internal-rewrite 2022-03-17T19:04:57.9359137Z * [new branch] ci-all/ci-vulkan-test -> origin/ci-all/ci-vulkan-test 2022-03-17T19:04:57.9360362Z * [new branch] ci-all/ci_vulkan_test -> origin/ci-all/ci_vulkan_test 2022-03-17T19:04:57.9361779Z * [new branch] ci-all/conv-double-backward-40999 -> origin/ci-all/conv-double-backward-40999 2022-03-17T19:04:57.9363170Z * [new branch] ci-all/conv3d-channels-last -> origin/ci-all/conv3d-channels-last 2022-03-17T19:04:57.9364565Z * [new branch] ci-all/cublas-triangular-solve -> origin/ci-all/cublas-triangular-solve 2022-03-17T19:04:57.9365919Z * [new branch] ci-all/cuda_111_debug_fix -> origin/ci-all/cuda_111_debug_fix 2022-03-17T19:04:57.9367359Z * [new branch] ci-all/cupti_so -> origin/ci-all/cupti_so 2022-03-17T19:04:57.9368555Z * [new branch] ci-all/cusolver-cholesky-1 -> origin/ci-all/cusolver-cholesky-1 2022-03-17T19:04:57.9370031Z * [new branch] ci-all/cusolver-cholesky-batched_cuda11.3 -> origin/ci-all/cusolver-cholesky-batched_cuda11.3 2022-03-17T19:04:57.9371270Z * [new branch] ci-all/cusolver-cholesky-inverse -> origin/ci-all/cusolver-cholesky-inverse 2022-03-17T19:04:57.9372777Z * [new branch] ci-all/cusolver-cholesky-solve -> origin/ci-all/cusolver-cholesky-solve 2022-03-17T19:04:57.9373910Z * [new branch] ci-all/cusolver-svd -> origin/ci-all/cusolver-svd 2022-03-17T19:04:57.9375262Z * [new branch] ci-all/debug-lmem -> origin/ci-all/debug-lmem 2022-03-17T19:04:57.9376585Z * [new branch] ci-all/debuggable-scheduled-ci -> origin/ci-all/debuggable-scheduled-ci 2022-03-17T19:04:57.9377985Z * [new branch] ci-all/default-dtype -> origin/ci-all/default-dtype 2022-03-17T19:04:57.9379226Z * [new branch] ci-all/dot_vdot_bmm_mv -> origin/ci-all/dot_vdot_bmm_mv 2022-03-17T19:04:57.9380605Z * [new branch] ci-all/enable_pe -> origin/ci-all/enable_pe 2022-03-17T19:04:57.9381868Z * [new branch] ci-all/fbjni_022 -> origin/ci-all/fbjni_022 2022-03-17T19:04:57.9383410Z * [new branch] ci-all/fbjni_022_2 -> origin/ci-all/fbjni_022_2 2022-03-17T19:04:57.9384726Z * [new branch] ci-all/fbjni_022_prefab_2 -> origin/ci-all/fbjni_022_prefab_2 2022-03-17T19:04:57.9386051Z * [new branch] ci-all/fe_test1 -> origin/ci-all/fe_test1 2022-03-17T19:04:57.9387286Z * [new branch] ci-all/fe_test1_fail -> origin/ci-all/fe_test1_fail 2022-03-17T19:04:57.9388692Z * [new branch] ci-all/fe_test1_fail_2 -> origin/ci-all/fe_test1_fail_2 2022-03-17T19:04:57.9389967Z * [new branch] ci-all/fe_test1_fail_2_my -> origin/ci-all/fe_test1_fail_2_my 2022-03-17T19:04:57.9391288Z * [new branch] ci-all/fe_test1_fail_2_my_m -> origin/ci-all/fe_test1_fail_2_my_m 2022-03-17T19:04:57.9392597Z * [new branch] ci-all/fix-54381-and-53976 -> origin/ci-all/fix-54381-and-53976 2022-03-17T19:04:57.9393955Z * [new branch] ci-all/fix-worksize-lstsq-magma -> origin/ci-all/fix-worksize-lstsq-magma 2022-03-17T19:04:57.9395215Z * [new branch] ci-all/fix_cu112_windows -> origin/ci-all/fix_cu112_windows 2022-03-17T19:04:57.9396510Z * [new branch] ci-all/fix_cuda112_jobs -> origin/ci-all/fix_cuda112_jobs 2022-03-17T19:04:57.9397729Z * [new branch] ci-all/gen-py -> origin/ci-all/gen-py 2022-03-17T19:04:57.9399933Z * [new branch] ci-all/gh/smessmer/307/head -> origin/ci-all/gh/smessmer/307/head 2022-03-17T19:04:57.9401182Z * [new branch] ci-all/h-huang -> origin/ci-all/h-huang 2022-03-17T19:04:57.9402924Z * [new branch] ci-all/hackathon_opinfo_matmul_2 -> origin/ci-all/hackathon_opinfo_matmul_2 2022-03-17T19:04:57.9404103Z * [new branch] ci-all/have-ci-all-still-work -> origin/ci-all/have-ci-all-still-work 2022-03-17T19:04:57.9405465Z * [new branch] ci-all/ic_prof_ref -> origin/ci-all/ic_prof_ref 2022-03-17T19:04:57.9406835Z * [new branch] ci-all/kineto_cpu -> origin/ci-all/kineto_cpu 2022-03-17T19:04:57.9408252Z * [new branch] ci-all/kineto_cpu_0303 -> origin/ci-all/kineto_cpu_0303 2022-03-17T19:04:57.9409531Z * [new branch] ci-all/kineto_exp -> origin/ci-all/kineto_exp 2022-03-17T19:04:57.9411413Z * [new branch] ci-all/krovatkin/profiling_fullforce -> origin/ci-all/krovatkin/profiling_fullforce 2022-03-17T19:04:57.9412985Z * [new branch] ci-all/krshrimali/record-time-slowtests -> origin/ci-all/krshrimali/record-time-slowtests 2022-03-17T19:04:57.9414283Z * [new branch] ci-all/mrshen -> origin/ci-all/mrshen 2022-03-17T19:04:57.9415584Z * [new branch] ci-all/mrshenli -> origin/ci-all/mrshenli 2022-03-17T19:04:57.9417051Z * [new branch] ci-all/mvz-ci-all -> origin/ci-all/mvz-ci-all 2022-03-17T19:04:57.9418474Z * [new branch] ci-all/mvz-te-ext-calls -> origin/ci-all/mvz-te-ext-calls 2022-03-17T19:04:57.9420464Z * [new branch] ci-all/orig/release/1.6 -> origin/ci-all/orig/release/1.6 2022-03-17T19:04:57.9421892Z * [new branch] ci-all/orig/release/1.7.1 -> origin/ci-all/orig/release/1.7.1 2022-03-17T19:04:57.9423654Z * [new branch] ci-all/peterbell10/batch-norm-cuda -> origin/ci-all/peterbell10/batch-norm-cuda 2022-03-17T19:04:57.9425033Z * [new branch] ci-all/post_backward_sync -> origin/ci-all/post_backward_sync 2022-03-17T19:04:57.9426476Z * [new branch] ci-all/pr_49201_2 -> origin/ci-all/pr_49201_2 2022-03-17T19:04:57.9428398Z * [new branch] ci-all/pritam/dist_autograd_sync -> origin/ci-all/pritam/dist_autograd_sync 2022-03-17T19:04:57.9429766Z * [new branch] ci-all/pritam/dist_autograd_sync_lastest -> origin/ci-all/pritam/dist_autograd_sync_lastest 2022-03-17T19:04:57.9431001Z * [new branch] ci-all/record-time-slowtests -> origin/ci-all/record-time-slowtests 2022-03-17T19:04:57.9432627Z * [new branch] ci-all/reenable-skipped-tests-for-other-cuda -> origin/ci-all/reenable-skipped-tests-for-other-cuda 2022-03-17T19:04:57.9433928Z * [new branch] ci-all/reenable-windows-standalone -> origin/ci-all/reenable-windows-standalone 2022-03-17T19:04:57.9435412Z * [new branch] ci-all/reenable-windows-test_nn-tests -> origin/ci-all/reenable-windows-test_nn-tests 2022-03-17T19:04:57.9437145Z * [new branch] ci-all/reland/develop/complex/masked_fill -> origin/ci-all/reland/develop/complex/masked_fill 2022-03-17T19:04:57.9438411Z * [new branch] ci-all/reland_addbmm -> origin/ci-all/reland_addbmm 2022-03-17T19:04:57.9439782Z * [new branch] ci-all/remove-thrust_t-arith -> origin/ci-all/remove-thrust_t-arith 2022-03-17T19:04:57.9441163Z * [new branch] ci-all/remove-thrust_t-eqne -> origin/ci-all/remove-thrust_t-eqne 2022-03-17T19:04:57.9442739Z * [new branch] ci-all/replace-linux-ci-11.1-with-11.2 -> origin/ci-all/replace-linux-ci-11.1-with-11.2 2022-03-17T19:04:57.9444915Z * [new branch] ci-all/replace-windows-11.1-ci-with-11.2 -> origin/ci-all/replace-windows-11.1-ci-with-11.2 2022-03-17T19:04:57.9446051Z * [new branch] ci-all/revert-render-test -> origin/ci-all/revert-render-test 2022-03-17T19:04:57.9447520Z * [new branch] ci-all/rohan/flaky -> origin/ci-all/rohan/flaky 2022-03-17T19:04:57.9448801Z * [new branch] ci-all/rohan/gputest -> origin/ci-all/rohan/gputest 2022-03-17T19:04:57.9450495Z * [new branch] ci-all/rohan/pg_wrapper -> origin/ci-all/rohan/pg_wrapper 2022-03-17T19:04:57.9451862Z * [new branch] ci-all/rohan/pg_wrapper_2 -> origin/ci-all/rohan/pg_wrapper_2 2022-03-17T19:04:57.9453285Z * [new branch] ci-all/rohan/pg_wrapper_ci_2 -> origin/ci-all/rohan/pg_wrapper_ci_2 2022-03-17T19:04:57.9454511Z * [new branch] ci-all/rohan/test_autograd_prof -> origin/ci-all/rohan/test_autograd_prof 2022-03-17T19:04:57.9456003Z * [new branch] ci-all/rohan/test_prof -> origin/ci-all/rohan/test_prof 2022-03-17T19:04:57.9457595Z * [new branch] ci-all/rvarm1/ci-all-barrier -> origin/ci-all/rvarm1/ci-all-barrier 2022-03-17T19:04:57.9459082Z * [new branch] ci-all/rvarm1/ci_all_ddp_mult_bwd -> origin/ci-all/rvarm1/ci_all_ddp_mult_bwd 2022-03-17T19:04:57.9460253Z * [new branch] ci-all/rvarm1/ddp_bwd -> origin/ci-all/rvarm1/ddp_bwd 2022-03-17T19:04:57.9462132Z * [new branch] ci-all/rvarm1/ddp_local_fix -> origin/ci-all/rvarm1/ddp_local_fix 2022-03-17T19:04:57.9463364Z * [new branch] ci-all/rvarm1/ddp_multi -> origin/ci-all/rvarm1/ddp_multi 2022-03-17T19:04:57.9468790Z * [new branch] ci-all/rvarm1/dist_optim -> origin/ci-all/rvarm1/dist_optim 2022-03-17T19:04:57.9470315Z * [new branch] ci-all/rvarm1/fix_unused -> origin/ci-all/rvarm1/fix_unused 2022-03-17T19:04:57.9471743Z * [new branch] ci-all/rvarm1/param_name_ddp -> origin/ci-all/rvarm1/param_name_ddp 2022-03-17T19:04:57.9472927Z * [new branch] ci-all/rvarm1/test_script -> origin/ci-all/rvarm1/test_script 2022-03-17T19:04:57.9474082Z * [new branch] ci-all/rvarm1/torch_check -> origin/ci-all/rvarm1/torch_check 2022-03-17T19:04:57.9475335Z * [new branch] ci-all/rvarm1/torch_check_ci_all -> origin/ci-all/rvarm1/torch_check_ci_all 2022-03-17T19:04:57.9476694Z * [new branch] ci-all/shared-cupti -> origin/ci-all/shared-cupti 2022-03-17T19:04:57.9478247Z * [new branch] ci-all/syncbn-channels-lastt-46906 -> origin/ci-all/syncbn-channels-lastt-46906 2022-03-17T19:04:57.9479578Z * [new branch] ci-all/test -> origin/ci-all/test 2022-03-17T19:04:57.9480932Z * [new branch] ci-all/test-time-do-not-merge -> origin/ci-all/test-time-do-not-merge 2022-03-17T19:04:57.9482439Z * [new branch] ci-all/test_all -> origin/ci-all/test_all 2022-03-17T19:04:57.9483763Z * [new branch] ci-all/test_binary_magma_2.5.4 -> origin/ci-all/test_binary_magma_2.5.4 2022-03-17T19:04:57.9485193Z * [new branch] ci-all/test_code_analyzer_regex -> origin/ci-all/test_code_analyzer_regex 2022-03-17T19:04:57.9486537Z * [new branch] ci-all/test_distutils -> origin/ci-all/test_distutils 2022-03-17T19:04:57.9488209Z * [new branch] ci-all/test_early_terminate_cuda_impact_on_test_times -> origin/ci-all/test_early_terminate_cuda_impact_on_test_times 2022-03-17T19:04:57.9489777Z * [new branch] ci-all/test_redispatching -> origin/ci-all/test_redispatching 2022-03-17T19:04:57.9491191Z * [new branch] ci-all/test_terminate_handler -> origin/ci-all/test_terminate_handler 2022-03-17T19:04:57.9492480Z * [new branch] ci-all/torchdeploy-reenable -> origin/ci-all/torchdeploy-reenable 2022-03-17T19:04:57.9493621Z * [new branch] ci-all/upd_k_0609 -> origin/ci-all/upd_k_0609 2022-03-17T19:04:57.9494690Z * [new branch] ci-all/upd_kineto_0512 -> origin/ci-all/upd_kineto_0512 2022-03-17T19:04:57.9495759Z * [new branch] ci-all/use_kineto_2 -> origin/ci-all/use_kineto_2 2022-03-17T19:04:57.9496800Z * [new branch] ci-all/use_kineto_3 -> origin/ci-all/use_kineto_3 2022-03-17T19:04:57.9497851Z * [new branch] ci-all/use_kineto_4 -> origin/ci-all/use_kineto_4 2022-03-17T19:04:57.9498908Z * [new branch] ci-all/use_kineto_5 -> origin/ci-all/use_kineto_5 2022-03-17T19:04:57.9500199Z * [new branch] ci-all/use_kineto_6 -> origin/ci-all/use_kineto_6 2022-03-17T19:04:57.9501340Z * [new branch] ci-all/v0dro-cusolver-lu -> origin/ci-all/v0dro-cusolver-lu 2022-03-17T19:04:57.9502557Z * [new branch] ci-all/v0dro-cusolver-lu-solve -> origin/ci-all/v0dro-cusolver-lu-solve 2022-03-17T19:04:57.9503869Z * [new branch] ci-all/vkuzo/fake_quant_test_20210120 -> origin/ci-all/vkuzo/fake_quant_test_20210120 2022-03-17T19:04:57.9505220Z * [new branch] ci-all/vulkan-android-by-default-ci -> origin/ci-all/vulkan-android-by-default-ci 2022-03-17T19:04:57.9506621Z * [new branch] ci-all/vulkan-android-ci-default -> origin/ci-all/vulkan-android-ci-default 2022-03-17T19:04:57.9508088Z * [new branch] ci-all/walterddr/early_terminate_cuda_jit_take_2 -> origin/ci-all/walterddr/early_terminate_cuda_jit_take_2 2022-03-17T19:04:57.9509338Z * [new branch] ci-all/werror_test -> origin/ci-all/werror_test 2022-03-17T19:04:57.9510515Z * [new branch] ci-all/win_cupti -> origin/ci-all/win_cupti 2022-03-17T19:04:57.9511560Z * [new branch] ci-all/win_kineto -> origin/ci-all/win_kineto 2022-03-17T19:04:57.9512773Z * [new branch] ci-all/yanlizhao/enableStaticGraph -> origin/ci-all/yanlizhao/enableStaticGraph 2022-03-17T19:04:57.9514090Z * [new branch] ci-all/yanlizhao/moveRebuildBucket -> origin/ci-all/yanlizhao/moveRebuildBucket 2022-03-17T19:04:57.9515359Z * [new branch] ci-all/yanlizhao/reduceMemory -> origin/ci-all/yanlizhao/reduceMemory 2022-03-17T19:04:57.9516825Z * [new branch] ci-all/yanlizhao/refactorInitializeBucketView -> origin/ci-all/yanlizhao/refactorInitializeBucketView 2022-03-17T19:04:57.9518305Z * [new branch] ci-gradcheck/forward_ad_gradcheck_ci_all -> origin/ci-gradcheck/forward_ad_gradcheck_ci_all 2022-03-17T19:04:57.9519429Z * [new branch] cifix -> origin/cifix 2022-03-17T19:04:57.9520799Z * [new branch] ciflow-push-trigger -> origin/ciflow-push-trigger 2022-03-17T19:04:57.9522357Z * [new branch] circle_ci_master_to_main -> origin/circle_ci_master_to_main 2022-03-17T19:04:57.9523660Z * [new branch] clean_config -> origin/clean_config 2022-03-17T19:04:57.9525000Z * [new branch] clean_up_monkeytype -> origin/clean_up_monkeytype 2022-03-17T19:04:57.9526443Z * [new branch] cleanup-scalar -> origin/cleanup-scalar 2022-03-17T19:04:57.9527877Z * [new branch] cleanup_submodule -> origin/cleanup_submodule 2022-03-17T19:04:57.9529847Z * [new branch] clee2000/disable-jobs-when-owner-not-pytorch -> origin/clee2000/disable-jobs-when-owner-not-pytorch 2022-03-17T19:04:57.9531092Z * [new branch] cluster_test -> origin/cluster_test 2022-03-17T19:04:57.9532513Z * [new branch] codegen_external -> origin/codegen_external 2022-03-17T19:04:57.9533850Z * [new branch] combine-test-times -> origin/combine-test-times 2022-03-17T19:04:57.9535333Z * [new branch] complex_backward -> origin/complex_backward 2022-03-17T19:04:57.9536737Z * [new branch] complex_support -> origin/complex_support 2022-03-17T19:04:57.9538566Z * [new branch] condition_cuda -> origin/condition_cuda 2022-03-17T19:04:57.9539959Z * [new branch] consolidate_manywheel_build -> origin/consolidate_manywheel_build 2022-03-17T19:04:57.9541181Z * [new branch] constant -> origin/constant 2022-03-17T19:04:57.9542527Z * [new branch] conv_empty_batch -> origin/conv_empty_batch 2022-03-17T19:04:57.9543983Z * [new branch] cpp_advanced_indexing -> origin/cpp_advanced_indexing 2022-03-17T19:04:57.9545430Z * [new branch] cpp_embedding_enum -> origin/cpp_embedding_enum 2022-03-17T19:04:57.9546912Z * [new branch] cpp_extension_subprocess_encoding_fixes -> origin/cpp_extension_subprocess_encoding_fixes 2022-03-17T19:04:57.9548263Z * [new branch] cpp_fix_functional_options_naming -> origin/cpp_fix_functional_options_naming 2022-03-17T19:04:57.9549541Z * [new branch] cpp_functional_non_const_ref -> origin/cpp_functional_non_const_ref 2022-03-17T19:04:57.9550809Z * [new branch] cpp_functional_options -> origin/cpp_functional_options 2022-03-17T19:04:57.9552274Z * [new branch] cpp_functional_options_backup -> origin/cpp_functional_options_backup 2022-03-17T19:04:57.9553685Z * [new branch] cpp_functional_options_backup_patch1 -> origin/cpp_functional_options_backup_patch1 2022-03-17T19:04:57.9554954Z * [new branch] cpp_functional_options_backup_patch2 -> origin/cpp_functional_options_backup_patch2 2022-03-17T19:04:57.9556169Z * [new branch] cpp_functional_options_v2 -> origin/cpp_functional_options_v2 2022-03-17T19:04:57.9557428Z * [new branch] cpp_hooks -> origin/cpp_hooks 2022-03-17T19:04:57.9558950Z * [new branch] cpp_module_options_arg_optional -> origin/cpp_module_options_arg_optional 2022-03-17T19:04:57.9560167Z * [new branch] cpp_nn_layers -> origin/cpp_nn_layers 2022-03-17T19:04:57.9561676Z * [new branch] cpp_parity_test_torch_nn_modules -> origin/cpp_parity_test_torch_nn_modules 2022-03-17T19:04:57.9563511Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing -> origin/cpp_tensor_default_dtype_and_advanced_indexing 2022-03-17T19:04:57.9565336Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_backup -> origin/cpp_tensor_default_dtype_and_advanced_indexing_backup 2022-03-17T19:04:57.9566836Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_new -> origin/cpp_tensor_default_dtype_and_advanced_indexing_new 2022-03-17T19:04:57.9568338Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_new_exp1_backup -> origin/cpp_tensor_default_dtype_and_advanced_indexing_new_exp1_backup 2022-03-17T19:04:57.9569795Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_tmp_exp -> origin/cpp_tensor_default_dtype_and_advanced_indexing_tmp_exp 2022-03-17T19:04:57.9571041Z * [new branch] cpp_tensor_default_dtype_for_integer -> origin/cpp_tensor_default_dtype_for_integer 2022-03-17T19:04:57.9572052Z * [new branch] cpp_tensor_indexing -> origin/cpp_tensor_indexing 2022-03-17T19:04:57.9573132Z * [new branch] cpu_fastrnn -> origin/cpu_fastrnn 2022-03-17T19:04:57.9574315Z * [new branch] create-ucc-pg -> origin/create-ucc-pg 2022-03-17T19:04:57.9575693Z * [new branch] cuda-long-matmul -> origin/cuda-long-matmul 2022-03-17T19:04:57.9577242Z * [new branch] cuda11_nightlies -> origin/cuda11_nightlies 2022-03-17T19:04:57.9578584Z * [new branch] cuda_stream_jit -> origin/cuda_stream_jit 2022-03-17T19:04:57.9579987Z * [new branch] cudahookbuilderror -> origin/cudahookbuilderror 2022-03-17T19:04:57.9581681Z * [new branch] cudnn-conv-refactor -> origin/cudnn-conv-refactor 2022-03-17T19:04:57.9583066Z * [new branch] cudnn-workspace-limit -> origin/cudnn-workspace-limit 2022-03-17T19:04:57.9584399Z * [new branch] cummax -> origin/cummax 2022-03-17T19:04:57.9585832Z * [new branch] cummax_cummin -> origin/cummax_cummin 2022-03-17T19:04:57.9587267Z * [new branch] cumulative_trapezoid_review -> origin/cumulative_trapezoid_review 2022-03-17T19:04:57.9589346Z * [new branch] dagitses/#62566/optional-delegate-cmp -> origin/dagitses/#62566/optional-delegate-cmp 2022-03-17T19:04:57.9591049Z * [new branch] davidberard98/ltc-docs -> origin/davidberard98/ltc-docs 2022-03-17T19:04:57.9592309Z * [new branch] davidberard98/ubsan-simpleireval -> origin/davidberard98/ubsan-simpleireval 2022-03-17T19:04:57.9593615Z * [new branch] dbr-conv -> origin/dbr-conv 2022-03-17T19:04:57.9595087Z * [new branch] ddp_with_rpc_unit_test -> origin/ddp_with_rpc_unit_test 2022-03-17T19:04:57.9596320Z * [new branch] debug-cpu -> origin/debug-cpu 2022-03-17T19:04:57.9598235Z * [new branch] debugLvlPull -> origin/debugLvlPull 2022-03-17T19:04:57.9599216Z * [new branch] debugging -> origin/debugging 2022-03-17T19:04:57.9600631Z * [new branch] dec14_cuda_stream -> origin/dec14_cuda_stream 2022-03-17T19:04:57.9602042Z * [new branch] default_argument -> origin/default_argument 2022-03-17T19:04:57.9603468Z * [new branch] default_hooks -> origin/default_hooks 2022-03-17T19:04:57.9604980Z * [new branch] devcon -> origin/devcon 2022-03-17T19:04:57.9606260Z * [new branch] devcontainer -> origin/devcontainer 2022-03-17T19:04:57.9607690Z * [new branch] device_type_testing -> origin/device_type_testing 2022-03-17T19:04:57.9609687Z * [new branch] deyu/remove_using_external_data_format -> origin/deyu/remove_using_external_data_format 2022-03-17T19:04:57.9610763Z * [new branch] dfs-needed -> origin/dfs-needed 2022-03-17T19:04:57.9612116Z * [new branch] diag-embed -> origin/diag-embed 2022-03-17T19:04:57.9613440Z * [new branch] diff1 -> origin/diff1 2022-03-17T19:04:57.9614904Z * [new branch] dirty_nnc_randomized -> origin/dirty_nnc_randomized 2022-03-17T19:04:57.9616197Z * [new branch] disable-test -> origin/disable-test 2022-03-17T19:04:57.9617653Z * [new branch] disable-test-history -> origin/disable-test-history 2022-03-17T19:04:57.9619116Z * [new branch] disable_peer_verify_nodesource -> origin/disable_peer_verify_nodesource 2022-03-17T19:04:57.9620580Z * [new branch] dist-documentation-patch-1.8 -> origin/dist-documentation-patch-1.8 2022-03-17T19:04:57.9621850Z * [new branch] dist-expand-refactor -> origin/dist-expand-refactor 2022-03-17T19:04:57.9623124Z * [new branch] dist-grad -> origin/dist-grad 2022-03-17T19:04:57.9624495Z * [new branch] dist_codeowners -> origin/dist_codeowners 2022-03-17T19:04:57.9625845Z * [new branch] dl_compiler_101 -> origin/dl_compiler_101 2022-03-17T19:04:57.9627105Z * [new branch] doc -> origin/doc 2022-03-17T19:04:57.9628502Z * [new branch] doc_fix -> origin/doc_fix 2022-03-17T19:04:57.9629924Z * [new branch] doc_monitored_barrier -> origin/doc_monitored_barrier 2022-03-17T19:04:57.9631371Z * [new branch] docker-job-chown-workspace -> origin/docker-job-chown-workspace 2022-03-17T19:04:57.9632767Z * [new branch] docs-fix-forward_ad-make_dual -> origin/docs-fix-forward_ad-make_dual 2022-03-17T19:04:57.9634023Z * [new branch] dont_decompose -> origin/dont_decompose 2022-03-17T19:04:57.9636008Z * [new branch] driazati/2test2/1 -> origin/driazati/2test2/1 2022-03-17T19:04:57.9637155Z * [new branch] driazati/2test2/2 -> origin/driazati/2test2/2 2022-03-17T19:04:57.9638450Z * [new branch] driazati/awsretry -> origin/driazati/awsretry 2022-03-17T19:04:57.9639751Z * [new branch] driazati/bp -> origin/driazati/bp 2022-03-17T19:04:57.9641066Z * [new branch] driazati/breakpad -> origin/driazati/breakpad 2022-03-17T19:04:57.9642810Z * [new branch] driazati/breakpad2 -> origin/driazati/breakpad2 2022-03-17T19:04:57.9644425Z * [new branch] driazati/breakpad_cmake -> origin/driazati/breakpad_cmake 2022-03-17T19:04:57.9645648Z * [new branch] driazati/breakpad_cmake2 -> origin/driazati/breakpad_cmake2 2022-03-17T19:04:57.9646936Z * [new branch] driazati/circle -> origin/driazati/circle 2022-03-17T19:04:57.9648229Z * [new branch] driazati/circle2 -> origin/driazati/circle2 2022-03-17T19:04:57.9649620Z * [new branch] driazati/doc_previews_fix -> origin/driazati/doc_previews_fix 2022-03-17T19:04:57.9651465Z * [new branch] driazati/docs/12 -> origin/driazati/docs/12 2022-03-17T19:04:57.9656324Z * [new branch] driazati/docsdet -> origin/driazati/docsdet 2022-03-17T19:04:57.9656764Z * [new branch] driazati/end -> origin/driazati/end 2022-03-17T19:04:57.9658931Z * [new branch] driazati/fastcov -> origin/driazati/fastcov 2022-03-17T19:04:57.9659527Z * [new branch] driazati/fixwarn2 -> origin/driazati/fixwarn2 2022-03-17T19:04:57.9660069Z * [new branch] driazati/focal -> origin/driazati/focal 2022-03-17T19:04:57.9660637Z * [new branch] driazati/github_pin -> origin/driazati/github_pin 2022-03-17T19:04:57.9661183Z * [new branch] driazati/it -> origin/driazati/it 2022-03-17T19:04:57.9661780Z * [new branch] driazati/lazy2 -> origin/driazati/lazy2 2022-03-17T19:04:57.9663861Z * [new branch] driazati/noart -> origin/driazati/noart 2022-03-17T19:04:57.9665883Z * [new branch] driazati/rdsre -> origin/driazati/rdsre 2022-03-17T19:04:57.9667390Z * [new branch] driazati/regenfix -> origin/driazati/regenfix 2022-03-17T19:04:57.9668838Z * [new branch] driazati/reme -> origin/driazati/reme 2022-03-17T19:04:57.9670230Z * [new branch] driazati/remove_bd -> origin/driazati/remove_bd 2022-03-17T19:04:57.9671735Z * [new branch] driazati/remove_breakpad -> origin/driazati/remove_breakpad 2022-03-17T19:04:57.9673087Z * [new branch] driazati/remove_render -> origin/driazati/remove_render 2022-03-17T19:04:57.9674254Z * [new branch] driazati/scc -> origin/driazati/scc 2022-03-17T19:04:57.9675461Z * [new branch] driazati/serialize_api -> origin/driazati/serialize_api 2022-03-17T19:04:57.9676660Z * [new branch] driazati/tdp -> origin/driazati/tdp 2022-03-17T19:04:57.9677847Z * [new branch] driazati/test_rds_proxy -> origin/driazati/test_rds_proxy 2022-03-17T19:04:57.9679043Z * [new branch] driazati/testbranch -> origin/driazati/testbranch 2022-03-17T19:04:57.9680245Z * [new branch] driazati/tester/2 -> origin/driazati/tester/2 2022-03-17T19:04:57.9681477Z * [new branch] driazati/testfailures -> origin/driazati/testfailures 2022-03-17T19:04:57.9683166Z * [new branch] driazati/update_onnx -> origin/driazati/update_onnx 2022-03-17T19:04:57.9684501Z * [new branch] driazati/windows_s3_reports -> origin/driazati/windows_s3_reports 2022-03-17T19:04:57.9685735Z * [new branch] duplicate_of_56029 -> origin/duplicate_of_56029 2022-03-17T19:04:57.9686947Z * [new branch] dynamic_reduce_range -> origin/dynamic_reduce_range 2022-03-17T19:04:57.9688096Z * [new branch] dynamic_shapes -> origin/dynamic_shapes 2022-03-17T19:04:57.9689455Z * [new branch] dynamic_shapes_cpp -> origin/dynamic_shapes_cpp 2022-03-17T19:04:57.9690860Z * [new branch] dynamic_shapes_python -> origin/dynamic_shapes_python 2022-03-17T19:04:57.9692495Z * [new branch] eager_compilation -> origin/eager_compilation 2022-03-17T19:04:57.9693689Z * [new branch] ed02619 -> origin/ed02619 2022-03-17T19:04:57.9695195Z * [new branch] edward-io-patch-1 -> origin/edward-io-patch-1 2022-03-17T19:04:57.9696090Z * [new branch] elementwise_binary_ii_opinfos -> origin/elementwise_binary_ii_opinfos 2022-03-17T19:04:57.9697388Z * [new branch] elementwise_binary_opinfos -> origin/elementwise_binary_opinfos 2022-03-17T19:04:57.9699738Z * [new branch] embedding-cub-2 -> origin/embedding-cub-2 2022-03-17T19:04:57.9701074Z * [new branch] empty -> origin/empty 2022-03-17T19:04:57.9702203Z * [new branch] enable_custom_codefilter -> origin/enable_custom_codefilter 2022-03-17T19:04:57.9703506Z * [new branch] enable_metal_in_pytorch_macos_nightly -> origin/enable_metal_in_pytorch_macos_nightly 2022-03-17T19:04:57.9705450Z * [new branch] enabling_checking_typing_attributes_during_rcb_lookup -> origin/enabling_checking_typing_attributes_during_rcb_lookup 2022-03-17T19:04:57.9706859Z * [new branch] error_check_env_fix -> origin/error_check_env_fix 2022-03-17T19:04:57.9707994Z * [new branch] error_input_tests_to_opinfo -> origin/error_input_tests_to_opinfo 2022-03-17T19:04:57.9709269Z * [new branch] error_on_composite_registrations -> origin/error_on_composite_registrations 2022-03-17T19:04:57.9710448Z * [new branch] exercise1 -> origin/exercise1 2022-03-17T19:04:57.9711719Z * [new branch] export-D14992856 -> origin/export-D14992856 2022-03-17T19:04:57.9713146Z * [new branch] export-D14992857 -> origin/export-D14992857 2022-03-17T19:04:57.9714505Z * [new branch] export-D15634027 -> origin/export-D15634027 2022-03-17T19:04:57.9715817Z * [new branch] export-D15795024 -> origin/export-D15795024 2022-03-17T19:04:57.9717306Z * [new branch] export-D15814743 -> origin/export-D15814743 2022-03-17T19:04:57.9718801Z * [new branch] expr_handle_pow -> origin/expr_handle_pow 2022-03-17T19:04:57.9719961Z * [new branch] fb-config -> origin/fb-config 2022-03-17T19:04:57.9721630Z * [new branch] fbcode/warm -> origin/fbcode/warm 2022-03-17T19:04:57.9725294Z * [new branch] fbjni_022 -> origin/fbjni_022 2022-03-17T19:04:57.9726413Z * [new branch] fbjni_022_2 -> origin/fbjni_022_2 2022-03-17T19:04:57.9727734Z * [new branch] fbjni_022_prefab -> origin/fbjni_022_prefab 2022-03-17T19:04:57.9728959Z * [new branch] fbjni_022_prefab_2 -> origin/fbjni_022_prefab_2 2022-03-17T19:04:57.9730192Z * [new branch] fbsync -> origin/fbsync 2022-03-17T19:04:57.9731523Z * [new branch] fix-cat-transform-docs -> origin/fix-cat-transform-docs 2022-03-17T19:04:57.9732875Z * [new branch] fix-dirichlet-shape -> origin/fix-dirichlet-shape 2022-03-17T19:04:57.9734324Z * [new branch] fix-mac-test-report -> origin/fix-mac-test-report 2022-03-17T19:04:57.9735781Z * [new branch] fix-mac-uploading -> origin/fix-mac-uploading 2022-03-17T19:04:57.9737204Z * [new branch] fix-optim-annotation -> origin/fix-optim-annotation 2022-03-17T19:04:57.9738659Z * [new branch] fix-timer-overflow -> origin/fix-timer-overflow 2022-03-17T19:04:57.9739927Z * [new branch] fix-tools -> origin/fix-tools 2022-03-17T19:04:57.9741251Z * [new branch] fix_conv -> origin/fix_conv 2022-03-17T19:04:57.9742635Z * [new branch] fix_cpp_impl_parity -> origin/fix_cpp_impl_parity 2022-03-17T19:04:57.9744161Z * [new branch] fix_ddp_doc -> origin/fix_ddp_doc 2022-03-17T19:04:57.9745596Z * [new branch] fix_dispatch -> origin/fix_dispatch 2022-03-17T19:04:57.9746881Z * [new branch] fix_doc_issues -> origin/fix_doc_issues 2022-03-17T19:04:57.9748178Z * [new branch] fix_doc_new -> origin/fix_doc_new 2022-03-17T19:04:57.9749534Z * [new branch] fix_docs_Aprl -> origin/fix_docs_Aprl 2022-03-17T19:04:57.9750843Z * [new branch] fix_error_msg -> origin/fix_error_msg 2022-03-17T19:04:57.9752095Z * [new branch] fix_foreach_test -> origin/fix_foreach_test 2022-03-17T19:04:57.9753526Z * [new branch] fix_graphmodule_singleton -> origin/fix_graphmodule_singleton 2022-03-17T19:04:57.9754970Z * [new branch] fix_javadoc_url -> origin/fix_javadoc_url 2022-03-17T19:04:57.9756424Z * [new branch] fix_jit_overview_fut -> origin/fix_jit_overview_fut 2022-03-17T19:04:57.9758166Z * [new branch] fix_module_forward_optional_arg -> origin/fix_module_forward_optional_arg 2022-03-17T19:04:57.9759803Z * [new branch] fix_module_forward_optional_arg_backup -> origin/fix_module_forward_optional_arg_backup 2022-03-17T19:04:57.9761365Z * [new branch] fix_multidim_tensor_ctor -> origin/fix_multidim_tensor_ctor 2022-03-17T19:04:57.9762915Z * [new branch] fix_nccl -> origin/fix_nccl 2022-03-17T19:04:57.9764081Z * [new branch] fix_nccl_2 -> origin/fix_nccl_2 2022-03-17T19:04:57.9765393Z * [new branch] fix_portforward -> origin/fix_portforward 2022-03-17T19:04:57.9766712Z * [new branch] fix_shape_prop -> origin/fix_shape_prop 2022-03-17T19:04:57.9768241Z * [new branch] fix_tensorpipe_cmake_option -> origin/fix_tensorpipe_cmake_option 2022-03-17T19:04:57.9769740Z * [new branch] fix_torch_module_check -> origin/fix_torch_module_check 2022-03-17T19:04:57.9771148Z * [new branch] fix_torch_tensor_ctor_4th_try -> origin/fix_torch_tensor_ctor_4th_try 2022-03-17T19:04:57.9772676Z * [new branch] fix_tp_cma_short_read -> origin/fix_tp_cma_short_read 2022-03-17T19:04:57.9774057Z * [new branch] fixoptionalargs -> origin/fixoptionalargs 2022-03-17T19:04:57.9775423Z * [new branch] flatbuffer -> origin/flatbuffer 2022-03-17T19:04:57.9776768Z * [new branch] float_requant_q8vadd -> origin/float_requant_q8vadd 2022-03-17T19:04:57.9778206Z * [new branch] float_requant_q8vadd_2021 -> origin/float_requant_q8vadd_2021 2022-03-17T19:04:57.9779626Z * [new branch] float_zero_point_qparam_cpu -> origin/float_zero_point_qparam_cpu 2022-03-17T19:04:57.9781066Z * [new branch] flops_print_fix -> origin/flops_print_fix 2022-03-17T19:04:57.9782766Z * [new branch] forbid-inplace-modifi-packing -> origin/forbid-inplace-modifi-packing 2022-03-17T19:04:57.9784228Z * [new branch] foreach_wip -> origin/foreach_wip 2022-03-17T19:04:57.9785542Z * [new branch] forward-fix-bc -> origin/forward-fix-bc 2022-03-17T19:04:57.9786767Z * [new branch] foward -> origin/foward 2022-03-17T19:04:57.9788084Z * [new branch] freeze_module2 -> origin/freeze_module2 2022-03-17T19:04:57.9789497Z * [new branch] freeze_subgraphs -> origin/freeze_subgraphs 2022-03-17T19:04:57.9790728Z * [new branch] freezing -> origin/freezing 2022-03-17T19:04:57.9791980Z * [new branch] fsdp_meta -> origin/fsdp_meta 2022-03-17T19:04:57.9793398Z * [new branch] fx-lowering-refactor -> origin/fx-lowering-refactor 2022-03-17T19:04:57.9795051Z * [new branch] fx-lowering-refactor-2 -> origin/fx-lowering-refactor-2 2022-03-17T19:04:57.9796723Z * [new branch] fx-lowering-refactor-3 -> origin/fx-lowering-refactor-3 2022-03-17T19:04:57.9798116Z * [new branch] fx_buffers -> origin/fx_buffers 2022-03-17T19:04:57.9799415Z * [new branch] fx_codegen -> origin/fx_codegen 2022-03-17T19:04:57.9800742Z * [new branch] fx_opinfo_port -> origin/fx_opinfo_port 2022-03-17T19:04:57.9802461Z * [new branch] garymm/dummy -> origin/garymm/dummy 2022-03-17T19:04:57.9803740Z * [new branch] gatherTopK -> origin/gatherTopK 2022-03-17T19:04:57.9804973Z * [new branch] gelu -> origin/gelu 2022-03-17T19:04:57.9806237Z * [new branch] get-workflow-name -> origin/get-workflow-name 2022-03-17T19:04:57.9808419Z * [new branch] gh/1ntEgr8/1/base -> origin/gh/1ntEgr8/1/base 2022-03-17T19:04:57.9810228Z * [new branch] gh/1ntEgr8/1/head -> origin/gh/1ntEgr8/1/head 2022-03-17T19:04:57.9811554Z * [new branch] gh/1ntEgr8/1/orig -> origin/gh/1ntEgr8/1/orig 2022-03-17T19:04:57.9812934Z * [new branch] gh/1ntEgr8/11/base -> origin/gh/1ntEgr8/11/base 2022-03-17T19:04:57.9814347Z * [new branch] gh/1ntEgr8/11/head -> origin/gh/1ntEgr8/11/head 2022-03-17T19:04:57.9815351Z * [new branch] gh/1ntEgr8/11/orig -> origin/gh/1ntEgr8/11/orig 2022-03-17T19:04:57.9816674Z * [new branch] gh/1ntEgr8/17/base -> origin/gh/1ntEgr8/17/base 2022-03-17T19:04:57.9817920Z * [new branch] gh/1ntEgr8/17/head -> origin/gh/1ntEgr8/17/head 2022-03-17T19:04:57.9819247Z * [new branch] gh/1ntEgr8/17/orig -> origin/gh/1ntEgr8/17/orig 2022-03-17T19:04:57.9820887Z * [new branch] gh/1ntEgr8/18/base -> origin/gh/1ntEgr8/18/base 2022-03-17T19:04:57.9822287Z * [new branch] gh/1ntEgr8/18/head -> origin/gh/1ntEgr8/18/head 2022-03-17T19:04:57.9823525Z * [new branch] gh/1ntEgr8/18/orig -> origin/gh/1ntEgr8/18/orig 2022-03-17T19:04:57.9824953Z * [new branch] gh/1ntEgr8/2/base -> origin/gh/1ntEgr8/2/base 2022-03-17T19:04:57.9826338Z * [new branch] gh/1ntEgr8/2/head -> origin/gh/1ntEgr8/2/head 2022-03-17T19:04:57.9827765Z * [new branch] gh/1ntEgr8/2/orig -> origin/gh/1ntEgr8/2/orig 2022-03-17T19:04:57.9829417Z * [new branch] gh/1ntEgr8/20/base -> origin/gh/1ntEgr8/20/base 2022-03-17T19:04:57.9830782Z * [new branch] gh/1ntEgr8/20/head -> origin/gh/1ntEgr8/20/head 2022-03-17T19:04:57.9832118Z * [new branch] gh/1ntEgr8/20/orig -> origin/gh/1ntEgr8/20/orig 2022-03-17T19:04:57.9833720Z * [new branch] gh/1ntEgr8/22/base -> origin/gh/1ntEgr8/22/base 2022-03-17T19:04:57.9834998Z * [new branch] gh/1ntEgr8/22/head -> origin/gh/1ntEgr8/22/head 2022-03-17T19:04:57.9836347Z * [new branch] gh/1ntEgr8/22/orig -> origin/gh/1ntEgr8/22/orig 2022-03-17T19:04:57.9838035Z * [new branch] gh/1ntEgr8/24/base -> origin/gh/1ntEgr8/24/base 2022-03-17T19:04:57.9839333Z * [new branch] gh/1ntEgr8/24/head -> origin/gh/1ntEgr8/24/head 2022-03-17T19:04:57.9840806Z * [new branch] gh/1ntEgr8/24/orig -> origin/gh/1ntEgr8/24/orig 2022-03-17T19:04:57.9843002Z * [new branch] gh/1ntEgr8/25/base -> origin/gh/1ntEgr8/25/base 2022-03-17T19:04:57.9844304Z * [new branch] gh/1ntEgr8/25/head -> origin/gh/1ntEgr8/25/head 2022-03-17T19:04:57.9845633Z * [new branch] gh/1ntEgr8/25/orig -> origin/gh/1ntEgr8/25/orig 2022-03-17T19:04:57.9847127Z * [new branch] gh/1ntEgr8/3/base -> origin/gh/1ntEgr8/3/base 2022-03-17T19:04:57.9848571Z * [new branch] gh/1ntEgr8/3/head -> origin/gh/1ntEgr8/3/head 2022-03-17T19:04:57.9849792Z * [new branch] gh/1ntEgr8/3/orig -> origin/gh/1ntEgr8/3/orig 2022-03-17T19:04:57.9851865Z * [new branch] gh/Amyh11325/10/base -> origin/gh/Amyh11325/10/base 2022-03-17T19:04:57.9853337Z * [new branch] gh/Amyh11325/10/head -> origin/gh/Amyh11325/10/head 2022-03-17T19:04:57.9854695Z * [new branch] gh/Amyh11325/10/orig -> origin/gh/Amyh11325/10/orig 2022-03-17T19:04:57.9856630Z * [new branch] gh/Amyh11325/13/base -> origin/gh/Amyh11325/13/base 2022-03-17T19:04:57.9858059Z * [new branch] gh/Amyh11325/13/head -> origin/gh/Amyh11325/13/head 2022-03-17T19:04:57.9859389Z * [new branch] gh/Amyh11325/13/orig -> origin/gh/Amyh11325/13/orig 2022-03-17T19:04:57.9861038Z * [new branch] gh/Amyh11325/18/base -> origin/gh/Amyh11325/18/base 2022-03-17T19:04:57.9862584Z * [new branch] gh/Amyh11325/18/head -> origin/gh/Amyh11325/18/head 2022-03-17T19:04:57.9863960Z * [new branch] gh/Amyh11325/18/orig -> origin/gh/Amyh11325/18/orig 2022-03-17T19:04:57.9865895Z * [new branch] gh/AshkanAliabadi/1/base -> origin/gh/AshkanAliabadi/1/base 2022-03-17T19:04:57.9867359Z * [new branch] gh/AshkanAliabadi/1/head -> origin/gh/AshkanAliabadi/1/head 2022-03-17T19:04:57.9868716Z * [new branch] gh/AshkanAliabadi/1/orig -> origin/gh/AshkanAliabadi/1/orig 2022-03-17T19:04:57.9870360Z * [new branch] gh/AshkanAliabadi/30/base -> origin/gh/AshkanAliabadi/30/base 2022-03-17T19:04:57.9871790Z * [new branch] gh/AshkanAliabadi/30/head -> origin/gh/AshkanAliabadi/30/head 2022-03-17T19:04:57.9873173Z * [new branch] gh/AshkanAliabadi/30/orig -> origin/gh/AshkanAliabadi/30/orig 2022-03-17T19:04:57.9874678Z * [new branch] gh/AshkanAliabadi/31/base -> origin/gh/AshkanAliabadi/31/base 2022-03-17T19:04:57.9876020Z * [new branch] gh/AshkanAliabadi/31/head -> origin/gh/AshkanAliabadi/31/head 2022-03-17T19:04:57.9877399Z * [new branch] gh/AshkanAliabadi/31/orig -> origin/gh/AshkanAliabadi/31/orig 2022-03-17T19:04:57.9879033Z * [new branch] gh/AshkanAliabadi/32/base -> origin/gh/AshkanAliabadi/32/base 2022-03-17T19:04:57.9880590Z * [new branch] gh/AshkanAliabadi/32/head -> origin/gh/AshkanAliabadi/32/head 2022-03-17T19:04:57.9881700Z * [new branch] gh/AshkanAliabadi/32/orig -> origin/gh/AshkanAliabadi/32/orig 2022-03-17T19:04:57.9884287Z * [new branch] gh/AshkanAliabadi/33/base -> origin/gh/AshkanAliabadi/33/base 2022-03-17T19:04:57.9885430Z * [new branch] gh/AshkanAliabadi/33/head -> origin/gh/AshkanAliabadi/33/head 2022-03-17T19:04:57.9886744Z * [new branch] gh/AshkanAliabadi/33/orig -> origin/gh/AshkanAliabadi/33/orig 2022-03-17T19:04:57.9888549Z * [new branch] gh/AshkanAliabadi/38/base -> origin/gh/AshkanAliabadi/38/base 2022-03-17T19:04:57.9889782Z * [new branch] gh/AshkanAliabadi/38/head -> origin/gh/AshkanAliabadi/38/head 2022-03-17T19:04:57.9891135Z * [new branch] gh/AshkanAliabadi/38/orig -> origin/gh/AshkanAliabadi/38/orig 2022-03-17T19:04:57.9892836Z * [new branch] gh/AshkanAliabadi/4/base -> origin/gh/AshkanAliabadi/4/base 2022-03-17T19:04:57.9894623Z * [new branch] gh/AshkanAliabadi/46/base -> origin/gh/AshkanAliabadi/46/base 2022-03-17T19:04:57.9896241Z * [new branch] gh/AshkanAliabadi/46/head -> origin/gh/AshkanAliabadi/46/head 2022-03-17T19:04:57.9897645Z * [new branch] gh/AshkanAliabadi/5/base -> origin/gh/AshkanAliabadi/5/base 2022-03-17T19:04:57.9899687Z * [new branch] gh/AshkanAliabadi/50/base -> origin/gh/AshkanAliabadi/50/base 2022-03-17T19:04:57.9901554Z * [new branch] gh/AshkanAliabadi/50/head -> origin/gh/AshkanAliabadi/50/head 2022-03-17T19:04:57.9902920Z * [new branch] gh/AshkanAliabadi/50/orig -> origin/gh/AshkanAliabadi/50/orig 2022-03-17T19:04:57.9904398Z * [new branch] gh/AshkanAliabadi/54/base -> origin/gh/AshkanAliabadi/54/base 2022-03-17T19:04:57.9905630Z * [new branch] gh/AshkanAliabadi/54/head -> origin/gh/AshkanAliabadi/54/head 2022-03-17T19:04:57.9906914Z * [new branch] gh/AshkanAliabadi/54/orig -> origin/gh/AshkanAliabadi/54/orig 2022-03-17T19:04:57.9908432Z * [new branch] gh/AshkanAliabadi/57/base -> origin/gh/AshkanAliabadi/57/base 2022-03-17T19:04:57.9909547Z * [new branch] gh/AshkanAliabadi/57/head -> origin/gh/AshkanAliabadi/57/head 2022-03-17T19:04:57.9910908Z * [new branch] gh/AshkanAliabadi/57/orig -> origin/gh/AshkanAliabadi/57/orig 2022-03-17T19:04:57.9912621Z * [new branch] gh/AshkanAliabadi/59/base -> origin/gh/AshkanAliabadi/59/base 2022-03-17T19:04:57.9913992Z * [new branch] gh/AshkanAliabadi/59/head -> origin/gh/AshkanAliabadi/59/head 2022-03-17T19:04:57.9915506Z * [new branch] gh/AshkanAliabadi/59/orig -> origin/gh/AshkanAliabadi/59/orig 2022-03-17T19:04:57.9916939Z * [new branch] gh/AshkanAliabadi/6/base -> origin/gh/AshkanAliabadi/6/base 2022-03-17T19:04:57.9918547Z * [new branch] gh/AshkanAliabadi/60/base -> origin/gh/AshkanAliabadi/60/base 2022-03-17T19:04:57.9919930Z * [new branch] gh/AshkanAliabadi/60/head -> origin/gh/AshkanAliabadi/60/head 2022-03-17T19:04:57.9921249Z * [new branch] gh/AshkanAliabadi/60/orig -> origin/gh/AshkanAliabadi/60/orig 2022-03-17T19:04:57.9923520Z * [new branch] gh/AshkanAliabadi/64/base -> origin/gh/AshkanAliabadi/64/base 2022-03-17T19:04:57.9925073Z * [new branch] gh/AshkanAliabadi/64/head -> origin/gh/AshkanAliabadi/64/head 2022-03-17T19:04:57.9926658Z * [new branch] gh/AshkanAliabadi/64/orig -> origin/gh/AshkanAliabadi/64/orig 2022-03-17T19:04:57.9928248Z * [new branch] gh/AshkanAliabadi/65/base -> origin/gh/AshkanAliabadi/65/base 2022-03-17T19:04:57.9929362Z * [new branch] gh/AshkanAliabadi/65/head -> origin/gh/AshkanAliabadi/65/head 2022-03-17T19:04:57.9930605Z * [new branch] gh/AshkanAliabadi/65/orig -> origin/gh/AshkanAliabadi/65/orig 2022-03-17T19:04:57.9932311Z * [new branch] gh/AshkanAliabadi/66/base -> origin/gh/AshkanAliabadi/66/base 2022-03-17T19:04:57.9933843Z * [new branch] gh/AshkanAliabadi/66/head -> origin/gh/AshkanAliabadi/66/head 2022-03-17T19:04:57.9935174Z * [new branch] gh/AshkanAliabadi/66/orig -> origin/gh/AshkanAliabadi/66/orig 2022-03-17T19:04:57.9936869Z * [new branch] gh/AshkanAliabadi/67/base -> origin/gh/AshkanAliabadi/67/base 2022-03-17T19:04:57.9938192Z * [new branch] gh/AshkanAliabadi/67/head -> origin/gh/AshkanAliabadi/67/head 2022-03-17T19:04:57.9939640Z * [new branch] gh/AshkanAliabadi/67/orig -> origin/gh/AshkanAliabadi/67/orig 2022-03-17T19:04:57.9941254Z * [new branch] gh/AshkanAliabadi/68/base -> origin/gh/AshkanAliabadi/68/base 2022-03-17T19:04:57.9942788Z * [new branch] gh/AshkanAliabadi/68/head -> origin/gh/AshkanAliabadi/68/head 2022-03-17T19:04:57.9944045Z * [new branch] gh/AshkanAliabadi/68/orig -> origin/gh/AshkanAliabadi/68/orig 2022-03-17T19:04:57.9945789Z * [new branch] gh/BowenBao/109/base -> origin/gh/BowenBao/109/base 2022-03-17T19:04:57.9947229Z * [new branch] gh/BowenBao/109/head -> origin/gh/BowenBao/109/head 2022-03-17T19:04:57.9948622Z * [new branch] gh/BowenBao/109/orig -> origin/gh/BowenBao/109/orig 2022-03-17T19:04:57.9950235Z * [new branch] gh/BowenBao/110/base -> origin/gh/BowenBao/110/base 2022-03-17T19:04:57.9951530Z * [new branch] gh/BowenBao/110/head -> origin/gh/BowenBao/110/head 2022-03-17T19:04:57.9953256Z * [new branch] gh/BowenBao/111/base -> origin/gh/BowenBao/111/base 2022-03-17T19:04:57.9954510Z * [new branch] gh/BowenBao/111/head -> origin/gh/BowenBao/111/head 2022-03-17T19:04:57.9955864Z * [new branch] gh/BowenBao/112/base -> origin/gh/BowenBao/112/base 2022-03-17T19:04:57.9957223Z * [new branch] gh/BowenBao/112/head -> origin/gh/BowenBao/112/head 2022-03-17T19:04:57.9958681Z * [new branch] gh/BowenBao/113/base -> origin/gh/BowenBao/113/base 2022-03-17T19:04:57.9960095Z * [new branch] gh/BowenBao/113/head -> origin/gh/BowenBao/113/head 2022-03-17T19:04:57.9961596Z * [new branch] gh/BowenBao/114/base -> origin/gh/BowenBao/114/base 2022-03-17T19:04:57.9963169Z * [new branch] gh/BowenBao/114/head -> origin/gh/BowenBao/114/head 2022-03-17T19:04:57.9964768Z * [new branch] gh/BowenBao/115/base -> origin/gh/BowenBao/115/base 2022-03-17T19:04:57.9966247Z * [new branch] gh/BowenBao/115/head -> origin/gh/BowenBao/115/head 2022-03-17T19:04:57.9967698Z * [new branch] gh/BowenBao/116/base -> origin/gh/BowenBao/116/base 2022-03-17T19:04:57.9969012Z * [new branch] gh/BowenBao/116/head -> origin/gh/BowenBao/116/head 2022-03-17T19:04:57.9970663Z * [new branch] gh/BowenBao/117/base -> origin/gh/BowenBao/117/base 2022-03-17T19:04:57.9971797Z * [new branch] gh/BowenBao/117/head -> origin/gh/BowenBao/117/head 2022-03-17T19:04:57.9973288Z * [new branch] gh/BowenBao/118/base -> origin/gh/BowenBao/118/base 2022-03-17T19:04:57.9974630Z * [new branch] gh/BowenBao/118/head -> origin/gh/BowenBao/118/head 2022-03-17T19:04:57.9976164Z * [new branch] gh/BowenBao/119/base -> origin/gh/BowenBao/119/base 2022-03-17T19:04:57.9977627Z * [new branch] gh/BowenBao/119/head -> origin/gh/BowenBao/119/head 2022-03-17T19:04:57.9979005Z * [new branch] gh/BowenBao/120/base -> origin/gh/BowenBao/120/base 2022-03-17T19:04:57.9980394Z * [new branch] gh/BowenBao/120/head -> origin/gh/BowenBao/120/head 2022-03-17T19:04:57.9981915Z * [new branch] gh/BowenBao/121/base -> origin/gh/BowenBao/121/base 2022-03-17T19:04:57.9983221Z * [new branch] gh/BowenBao/121/head -> origin/gh/BowenBao/121/head 2022-03-17T19:04:57.9984791Z * [new branch] gh/BowenBao/122/base -> origin/gh/BowenBao/122/base 2022-03-17T19:04:57.9986060Z * [new branch] gh/BowenBao/122/head -> origin/gh/BowenBao/122/head 2022-03-17T19:04:57.9987490Z * [new branch] gh/BowenBao/123/base -> origin/gh/BowenBao/123/base 2022-03-17T19:04:57.9988911Z * [new branch] gh/BowenBao/123/head -> origin/gh/BowenBao/123/head 2022-03-17T19:04:57.9990454Z * [new branch] gh/BowenBao/124/base -> origin/gh/BowenBao/124/base 2022-03-17T19:04:57.9991668Z * [new branch] gh/BowenBao/124/head -> origin/gh/BowenBao/124/head 2022-03-17T19:04:57.9993218Z * [new branch] gh/BowenBao/125/base -> origin/gh/BowenBao/125/base 2022-03-17T19:04:57.9994689Z * [new branch] gh/BowenBao/125/head -> origin/gh/BowenBao/125/head 2022-03-17T19:04:57.9996182Z * [new branch] gh/BowenBao/126/base -> origin/gh/BowenBao/126/base 2022-03-17T19:04:57.9997586Z * [new branch] gh/BowenBao/126/head -> origin/gh/BowenBao/126/head 2022-03-17T19:04:57.9999118Z * [new branch] gh/BowenBao/127/base -> origin/gh/BowenBao/127/base 2022-03-17T19:04:58.0000402Z * [new branch] gh/BowenBao/127/head -> origin/gh/BowenBao/127/head 2022-03-17T19:04:58.0002143Z * [new branch] gh/BowenBao/128/base -> origin/gh/BowenBao/128/base 2022-03-17T19:04:58.0005261Z * [new branch] gh/BowenBao/128/head -> origin/gh/BowenBao/128/head 2022-03-17T19:04:58.0006565Z * [new branch] gh/BowenBao/129/base -> origin/gh/BowenBao/129/base 2022-03-17T19:04:58.0007895Z * [new branch] gh/BowenBao/129/head -> origin/gh/BowenBao/129/head 2022-03-17T19:04:58.0009107Z * [new branch] gh/BowenBao/130/base -> origin/gh/BowenBao/130/base 2022-03-17T19:04:58.0010428Z * [new branch] gh/BowenBao/130/head -> origin/gh/BowenBao/130/head 2022-03-17T19:04:58.0012024Z * [new branch] gh/BowenBao/131/base -> origin/gh/BowenBao/131/base 2022-03-17T19:04:58.0013275Z * [new branch] gh/BowenBao/131/head -> origin/gh/BowenBao/131/head 2022-03-17T19:04:58.0015236Z * [new branch] gh/BowenBao/148/base -> origin/gh/BowenBao/148/base 2022-03-17T19:04:58.0016137Z * [new branch] gh/BowenBao/148/head -> origin/gh/BowenBao/148/head 2022-03-17T19:04:58.0017635Z * [new branch] gh/BowenBao/148/orig -> origin/gh/BowenBao/148/orig 2022-03-17T19:04:58.0019361Z * [new branch] gh/BowenBao/149/base -> origin/gh/BowenBao/149/base 2022-03-17T19:04:58.0020682Z * [new branch] gh/BowenBao/149/head -> origin/gh/BowenBao/149/head 2022-03-17T19:04:58.0022041Z * [new branch] gh/BowenBao/149/orig -> origin/gh/BowenBao/149/orig 2022-03-17T19:04:58.0023725Z * [new branch] gh/BowenBao/150/base -> origin/gh/BowenBao/150/base 2022-03-17T19:04:58.0025262Z * [new branch] gh/BowenBao/150/head -> origin/gh/BowenBao/150/head 2022-03-17T19:04:58.0026625Z * [new branch] gh/BowenBao/150/orig -> origin/gh/BowenBao/150/orig 2022-03-17T19:04:58.0028315Z * [new branch] gh/BowenBao/151/base -> origin/gh/BowenBao/151/base 2022-03-17T19:04:58.0029684Z * [new branch] gh/BowenBao/151/orig -> origin/gh/BowenBao/151/orig 2022-03-17T19:04:58.0031575Z * [new branch] gh/DeepDriving/12/base -> origin/gh/DeepDriving/12/base 2022-03-17T19:04:58.0033114Z * [new branch] gh/DeepDriving/12/head -> origin/gh/DeepDriving/12/head 2022-03-17T19:04:58.0034467Z * [new branch] gh/DeepDriving/12/orig -> origin/gh/DeepDriving/12/orig 2022-03-17T19:04:58.0036115Z * [new branch] gh/DeepDriving/2/base -> origin/gh/DeepDriving/2/base 2022-03-17T19:04:58.0037496Z * [new branch] gh/DeepDriving/2/head -> origin/gh/DeepDriving/2/head 2022-03-17T19:04:58.0038923Z * [new branch] gh/DeepDriving/2/orig -> origin/gh/DeepDriving/2/orig 2022-03-17T19:04:58.0040382Z * [new branch] gh/DeepDriving/3/base -> origin/gh/DeepDriving/3/base 2022-03-17T19:04:58.0041645Z * [new branch] gh/DeepDriving/3/head -> origin/gh/DeepDriving/3/head 2022-03-17T19:04:58.0043162Z * [new branch] gh/DeepDriving/3/orig -> origin/gh/DeepDriving/3/orig 2022-03-17T19:04:58.0044579Z * [new branch] gh/DeepDriving/4/base -> origin/gh/DeepDriving/4/base 2022-03-17T19:04:58.0046014Z * [new branch] gh/DeepDriving/4/head -> origin/gh/DeepDriving/4/head 2022-03-17T19:04:58.0047388Z * [new branch] gh/DeepDriving/4/orig -> origin/gh/DeepDriving/4/orig 2022-03-17T19:04:58.0048838Z * [new branch] gh/DeepDriving/5/base -> origin/gh/DeepDriving/5/base 2022-03-17T19:04:58.0050105Z * [new branch] gh/DeepDriving/5/head -> origin/gh/DeepDriving/5/head 2022-03-17T19:04:58.0051417Z * [new branch] gh/DeepDriving/5/orig -> origin/gh/DeepDriving/5/orig 2022-03-17T19:04:58.0053045Z * [new branch] gh/DeepDriving/7/base -> origin/gh/DeepDriving/7/base 2022-03-17T19:04:58.0054451Z * [new branch] gh/DeepDriving/7/head -> origin/gh/DeepDriving/7/head 2022-03-17T19:04:58.0055936Z * [new branch] gh/DeepDriving/8/base -> origin/gh/DeepDriving/8/base 2022-03-17T19:04:58.0067284Z * [new branch] gh/DeepDriving/8/head -> origin/gh/DeepDriving/8/head 2022-03-17T19:04:58.0068221Z * [new branch] gh/DeepDriving/8/orig -> origin/gh/DeepDriving/8/orig 2022-03-17T19:04:58.0068877Z * [new branch] gh/DeepDriving/9/base -> origin/gh/DeepDriving/9/base 2022-03-17T19:04:58.0069462Z * [new branch] gh/DeepDriving/9/head -> origin/gh/DeepDriving/9/head 2022-03-17T19:04:58.0070230Z * [new branch] gh/DeepDriving/9/orig -> origin/gh/DeepDriving/9/orig 2022-03-17T19:04:58.0071051Z * [new branch] gh/Feey0/26/base -> origin/gh/Feey0/26/base 2022-03-17T19:04:58.0076421Z * [new branch] gh/Feey0/26/head -> origin/gh/Feey0/26/head 2022-03-17T19:04:58.0077780Z * [new branch] gh/Feey0/26/orig -> origin/gh/Feey0/26/orig 2022-03-17T19:04:58.0079064Z * [new branch] gh/Feey0/28/base -> origin/gh/Feey0/28/base 2022-03-17T19:04:58.0080216Z * [new branch] gh/Feey0/28/head -> origin/gh/Feey0/28/head 2022-03-17T19:04:58.0082070Z * [new branch] gh/Feey0/28/orig -> origin/gh/Feey0/28/orig 2022-03-17T19:04:58.0083162Z * [new branch] gh/Feey0/29/base -> origin/gh/Feey0/29/base 2022-03-17T19:04:58.0084258Z * [new branch] gh/Feey0/29/head -> origin/gh/Feey0/29/head 2022-03-17T19:04:58.0085361Z * [new branch] gh/Feey0/29/orig -> origin/gh/Feey0/29/orig 2022-03-17T19:04:58.0086370Z * [new branch] gh/Feey0/3/base -> origin/gh/Feey0/3/base 2022-03-17T19:04:58.0087612Z * [new branch] gh/Feey0/3/head -> origin/gh/Feey0/3/head 2022-03-17T19:04:58.0088772Z * [new branch] gh/Feey0/3/orig -> origin/gh/Feey0/3/orig 2022-03-17T19:04:58.0089798Z * [new branch] gh/H-Huang/21/base -> origin/gh/H-Huang/21/base 2022-03-17T19:04:58.0091042Z * [new branch] gh/H-Huang/21/head -> origin/gh/H-Huang/21/head 2022-03-17T19:04:58.0092015Z * [new branch] gh/H-Huang/21/orig -> origin/gh/H-Huang/21/orig 2022-03-17T19:04:58.0095283Z * [new branch] gh/H-Huang/25/base -> origin/gh/H-Huang/25/base 2022-03-17T19:04:58.0098178Z * [new branch] gh/H-Huang/25/head -> origin/gh/H-Huang/25/head 2022-03-17T19:04:58.0101123Z * [new branch] gh/H-Huang/25/orig -> origin/gh/H-Huang/25/orig 2022-03-17T19:04:58.0104922Z * [new branch] gh/H-Huang/36/base -> origin/gh/H-Huang/36/base 2022-03-17T19:04:58.0107794Z * [new branch] gh/H-Huang/36/head -> origin/gh/H-Huang/36/head 2022-03-17T19:04:58.0110862Z * [new branch] gh/H-Huang/36/orig -> origin/gh/H-Huang/36/orig 2022-03-17T19:04:58.0114722Z * [new branch] gh/H-Huang/39/base -> origin/gh/H-Huang/39/base 2022-03-17T19:04:58.0117826Z * [new branch] gh/H-Huang/39/head -> origin/gh/H-Huang/39/head 2022-03-17T19:04:58.0120601Z * [new branch] gh/H-Huang/39/orig -> origin/gh/H-Huang/39/orig 2022-03-17T19:04:58.0124831Z * [new branch] gh/H-Huang/40/base -> origin/gh/H-Huang/40/base 2022-03-17T19:04:58.0127936Z * [new branch] gh/H-Huang/40/head -> origin/gh/H-Huang/40/head 2022-03-17T19:04:58.0130864Z * [new branch] gh/H-Huang/40/orig -> origin/gh/H-Huang/40/orig 2022-03-17T19:04:58.0164642Z * [new branch] gh/H-Huang/41/base -> origin/gh/H-Huang/41/base 2022-03-17T19:04:58.0165730Z * [new branch] gh/H-Huang/41/head -> origin/gh/H-Huang/41/head 2022-03-17T19:04:58.0166782Z * [new branch] gh/H-Huang/41/orig -> origin/gh/H-Huang/41/orig 2022-03-17T19:04:58.0167725Z * [new branch] gh/H-Huang/43/base -> origin/gh/H-Huang/43/base 2022-03-17T19:04:58.0168839Z * [new branch] gh/H-Huang/43/head -> origin/gh/H-Huang/43/head 2022-03-17T19:04:58.0169819Z * [new branch] gh/H-Huang/43/orig -> origin/gh/H-Huang/43/orig 2022-03-17T19:04:58.0170916Z * [new branch] gh/H-Huang/44/base -> origin/gh/H-Huang/44/base 2022-03-17T19:04:58.0171894Z * [new branch] gh/H-Huang/44/head -> origin/gh/H-Huang/44/head 2022-03-17T19:04:58.0172852Z * [new branch] gh/H-Huang/44/orig -> origin/gh/H-Huang/44/orig 2022-03-17T19:04:58.0173759Z * [new branch] gh/H-Huang/47/base -> origin/gh/H-Huang/47/base 2022-03-17T19:04:58.0174844Z * [new branch] gh/H-Huang/47/head -> origin/gh/H-Huang/47/head 2022-03-17T19:04:58.0178235Z * [new branch] gh/H-Huang/47/orig -> origin/gh/H-Huang/47/orig 2022-03-17T19:04:58.0182231Z * [new branch] gh/H-Huang/48/base -> origin/gh/H-Huang/48/base 2022-03-17T19:04:58.0186007Z * [new branch] gh/H-Huang/48/head -> origin/gh/H-Huang/48/head 2022-03-17T19:04:58.0189598Z * [new branch] gh/H-Huang/48/orig -> origin/gh/H-Huang/48/orig 2022-03-17T19:04:58.0194174Z * [new branch] gh/H-Huang/49/base -> origin/gh/H-Huang/49/base 2022-03-17T19:04:58.0198049Z * [new branch] gh/H-Huang/49/head -> origin/gh/H-Huang/49/head 2022-03-17T19:04:58.0201645Z * [new branch] gh/H-Huang/49/orig -> origin/gh/H-Huang/49/orig 2022-03-17T19:04:58.0206302Z * [new branch] gh/H-Huang/50/base -> origin/gh/H-Huang/50/base 2022-03-17T19:04:58.0210334Z * [new branch] gh/H-Huang/50/head -> origin/gh/H-Huang/50/head 2022-03-17T19:04:58.0213764Z * [new branch] gh/H-Huang/50/orig -> origin/gh/H-Huang/50/orig 2022-03-17T19:04:58.0219099Z * [new branch] gh/H-Huang/51/base -> origin/gh/H-Huang/51/base 2022-03-17T19:04:58.0220295Z * [new branch] gh/H-Huang/51/head -> origin/gh/H-Huang/51/head 2022-03-17T19:04:58.0221302Z * [new branch] gh/H-Huang/51/orig -> origin/gh/H-Huang/51/orig 2022-03-17T19:04:58.0222375Z * [new branch] gh/H-Huang/52/base -> origin/gh/H-Huang/52/base 2022-03-17T19:04:58.0223397Z * [new branch] gh/H-Huang/52/head -> origin/gh/H-Huang/52/head 2022-03-17T19:04:58.0224371Z * [new branch] gh/H-Huang/52/orig -> origin/gh/H-Huang/52/orig 2022-03-17T19:04:58.0225624Z * [new branch] gh/HDCharles/3/base -> origin/gh/HDCharles/3/base 2022-03-17T19:04:58.0226804Z * [new branch] gh/HDCharles/3/head -> origin/gh/HDCharles/3/head 2022-03-17T19:04:58.0227903Z * [new branch] gh/HDCharles/3/orig -> origin/gh/HDCharles/3/orig 2022-03-17T19:04:58.0229161Z * [new branch] gh/HDCharles/4/base -> origin/gh/HDCharles/4/base 2022-03-17T19:04:58.0230402Z * [new branch] gh/HDCharles/4/head -> origin/gh/HDCharles/4/head 2022-03-17T19:04:58.0233319Z * [new branch] gh/HDCharles/4/orig -> origin/gh/HDCharles/4/orig 2022-03-17T19:04:58.0234552Z * [new branch] gh/HDCharles/40/base -> origin/gh/HDCharles/40/base 2022-03-17T19:04:58.0235692Z * [new branch] gh/HDCharles/40/head -> origin/gh/HDCharles/40/head 2022-03-17T19:04:58.0237038Z * [new branch] gh/HDCharles/40/orig -> origin/gh/HDCharles/40/orig 2022-03-17T19:04:58.0238389Z * [new branch] gh/HDCharles/41/base -> origin/gh/HDCharles/41/base 2022-03-17T19:04:58.0239506Z * [new branch] gh/HDCharles/41/head -> origin/gh/HDCharles/41/head 2022-03-17T19:04:58.0240670Z * [new branch] gh/HDCharles/41/orig -> origin/gh/HDCharles/41/orig 2022-03-17T19:04:58.0243994Z * [new branch] gh/HDCharles/42/base -> origin/gh/HDCharles/42/base 2022-03-17T19:04:58.0245286Z * [new branch] gh/HDCharles/42/head -> origin/gh/HDCharles/42/head 2022-03-17T19:04:58.0246594Z * [new branch] gh/HDCharles/42/orig -> origin/gh/HDCharles/42/orig 2022-03-17T19:04:58.0247658Z * [new branch] gh/HDCharles/43/base -> origin/gh/HDCharles/43/base 2022-03-17T19:04:58.0248711Z * [new branch] gh/HDCharles/43/head -> origin/gh/HDCharles/43/head 2022-03-17T19:04:58.0249869Z * [new branch] gh/HDCharles/43/orig -> origin/gh/HDCharles/43/orig 2022-03-17T19:04:58.0250926Z * [new branch] gh/HDCharles/45/base -> origin/gh/HDCharles/45/base 2022-03-17T19:04:58.0252227Z * [new branch] gh/HDCharles/45/head -> origin/gh/HDCharles/45/head 2022-03-17T19:04:58.0254017Z * [new branch] gh/HDCharles/45/orig -> origin/gh/HDCharles/45/orig 2022-03-17T19:04:58.0255156Z * [new branch] gh/HDCharles/46/base -> origin/gh/HDCharles/46/base 2022-03-17T19:04:58.0256429Z * [new branch] gh/HDCharles/46/head -> origin/gh/HDCharles/46/head 2022-03-17T19:04:58.0257896Z * [new branch] gh/HDCharles/46/orig -> origin/gh/HDCharles/46/orig 2022-03-17T19:04:58.0260214Z * [new branch] gh/HDCharles/47/base -> origin/gh/HDCharles/47/base 2022-03-17T19:04:58.0265228Z * [new branch] gh/HDCharles/47/head -> origin/gh/HDCharles/47/head 2022-03-17T19:04:58.0266444Z * [new branch] gh/HDCharles/47/orig -> origin/gh/HDCharles/47/orig 2022-03-17T19:04:58.0267502Z * [new branch] gh/HDCharles/48/base -> origin/gh/HDCharles/48/base 2022-03-17T19:04:58.0268679Z * [new branch] gh/HDCharles/48/head -> origin/gh/HDCharles/48/head 2022-03-17T19:04:58.0269714Z * [new branch] gh/HDCharles/48/orig -> origin/gh/HDCharles/48/orig 2022-03-17T19:04:58.0270837Z * [new branch] gh/HDCharles/49/base -> origin/gh/HDCharles/49/base 2022-03-17T19:04:58.0271983Z * [new branch] gh/HDCharles/49/head -> origin/gh/HDCharles/49/head 2022-03-17T19:04:58.0273044Z * [new branch] gh/HDCharles/49/orig -> origin/gh/HDCharles/49/orig 2022-03-17T19:04:58.0274244Z * [new branch] gh/HDCharles/50/base -> origin/gh/HDCharles/50/base 2022-03-17T19:04:58.0275348Z * [new branch] gh/HDCharles/50/head -> origin/gh/HDCharles/50/head 2022-03-17T19:04:58.0276406Z * [new branch] gh/HDCharles/50/orig -> origin/gh/HDCharles/50/orig 2022-03-17T19:04:58.0277564Z * [new branch] gh/HDCharles/51/base -> origin/gh/HDCharles/51/base 2022-03-17T19:04:58.0278706Z * [new branch] gh/HDCharles/51/head -> origin/gh/HDCharles/51/head 2022-03-17T19:04:58.0279794Z * [new branch] gh/HDCharles/51/orig -> origin/gh/HDCharles/51/orig 2022-03-17T19:04:58.0281651Z * [new branch] gh/HDCharles/52/base -> origin/gh/HDCharles/52/base 2022-03-17T19:04:58.0296441Z * [new branch] gh/HDCharles/52/head -> origin/gh/HDCharles/52/head 2022-03-17T19:04:58.0297818Z * [new branch] gh/HDCharles/52/orig -> origin/gh/HDCharles/52/orig 2022-03-17T19:04:58.0299017Z * [new branch] gh/HDCharles/53/base -> origin/gh/HDCharles/53/base 2022-03-17T19:04:58.0299957Z * [new branch] gh/HDCharles/53/head -> origin/gh/HDCharles/53/head 2022-03-17T19:04:58.0300915Z * [new branch] gh/HDCharles/53/orig -> origin/gh/HDCharles/53/orig 2022-03-17T19:04:58.0302042Z * [new branch] gh/HDCharles/54/base -> origin/gh/HDCharles/54/base 2022-03-17T19:04:58.0303209Z * [new branch] gh/HDCharles/54/head -> origin/gh/HDCharles/54/head 2022-03-17T19:04:58.0304563Z * [new branch] gh/HDCharles/54/orig -> origin/gh/HDCharles/54/orig 2022-03-17T19:04:58.0305901Z * [new branch] gh/HDCharles/55/base -> origin/gh/HDCharles/55/base 2022-03-17T19:04:58.0307205Z * [new branch] gh/HDCharles/55/head -> origin/gh/HDCharles/55/head 2022-03-17T19:04:58.0308376Z * [new branch] gh/HDCharles/55/orig -> origin/gh/HDCharles/55/orig 2022-03-17T19:04:58.0309397Z * [new branch] gh/HDCharles/56/base -> origin/gh/HDCharles/56/base 2022-03-17T19:04:58.0310456Z * [new branch] gh/HDCharles/56/head -> origin/gh/HDCharles/56/head 2022-03-17T19:04:58.0311524Z * [new branch] gh/HDCharles/56/orig -> origin/gh/HDCharles/56/orig 2022-03-17T19:04:58.0312442Z * [new branch] gh/HDCharles/57/base -> origin/gh/HDCharles/57/base 2022-03-17T19:04:58.0313565Z * [new branch] gh/HDCharles/57/head -> origin/gh/HDCharles/57/head 2022-03-17T19:04:58.0314568Z * [new branch] gh/HDCharles/57/orig -> origin/gh/HDCharles/57/orig 2022-03-17T19:04:58.0315668Z * [new branch] gh/HDCharles/58/base -> origin/gh/HDCharles/58/base 2022-03-17T19:04:58.0317020Z * [new branch] gh/HDCharles/58/head -> origin/gh/HDCharles/58/head 2022-03-17T19:04:58.0318045Z * [new branch] gh/HDCharles/58/orig -> origin/gh/HDCharles/58/orig 2022-03-17T19:04:58.0319178Z * [new branch] gh/HDCharles/59/base -> origin/gh/HDCharles/59/base 2022-03-17T19:04:58.0320411Z * [new branch] gh/HDCharles/59/head -> origin/gh/HDCharles/59/head 2022-03-17T19:04:58.0321490Z * [new branch] gh/HDCharles/59/orig -> origin/gh/HDCharles/59/orig 2022-03-17T19:04:58.0322673Z * [new branch] gh/HDCharles/60/base -> origin/gh/HDCharles/60/base 2022-03-17T19:04:58.0323684Z * [new branch] gh/HDCharles/60/head -> origin/gh/HDCharles/60/head 2022-03-17T19:04:58.0324706Z * [new branch] gh/HDCharles/60/orig -> origin/gh/HDCharles/60/orig 2022-03-17T19:04:58.0325782Z * [new branch] gh/HDCharles/61/base -> origin/gh/HDCharles/61/base 2022-03-17T19:04:58.0326837Z * [new branch] gh/HDCharles/61/head -> origin/gh/HDCharles/61/head 2022-03-17T19:04:58.0327814Z * [new branch] gh/HDCharles/61/orig -> origin/gh/HDCharles/61/orig 2022-03-17T19:04:58.0328948Z * [new branch] gh/HDCharles/9/base -> origin/gh/HDCharles/9/base 2022-03-17T19:04:58.0330176Z * [new branch] gh/HDCharles/9/head -> origin/gh/HDCharles/9/head 2022-03-17T19:04:58.0331586Z * [new branch] gh/HDCharles/9/orig -> origin/gh/HDCharles/9/orig 2022-03-17T19:04:58.0333634Z * [new branch] gh/HaomiaoLiu730/1/base -> origin/gh/HaomiaoLiu730/1/base 2022-03-17T19:04:58.0335072Z * [new branch] gh/HaomiaoLiu730/1/head -> origin/gh/HaomiaoLiu730/1/head 2022-03-17T19:04:58.0337049Z * [new branch] gh/IvanKobzarev/1/base -> origin/gh/IvanKobzarev/1/base 2022-03-17T19:04:58.0338397Z * [new branch] gh/IvanKobzarev/1/head -> origin/gh/IvanKobzarev/1/head 2022-03-17T19:04:58.0340083Z * [new branch] gh/IvanKobzarev/19/base -> origin/gh/IvanKobzarev/19/base 2022-03-17T19:04:58.0341421Z * [new branch] gh/IvanKobzarev/19/head -> origin/gh/IvanKobzarev/19/head 2022-03-17T19:04:58.0342594Z * [new branch] gh/IvanKobzarev/19/orig -> origin/gh/IvanKobzarev/19/orig 2022-03-17T19:04:58.0344200Z * [new branch] gh/IvanKobzarev/2/base -> origin/gh/IvanKobzarev/2/base 2022-03-17T19:04:58.0345522Z * [new branch] gh/IvanKobzarev/2/head -> origin/gh/IvanKobzarev/2/head 2022-03-17T19:04:58.0348200Z * [new branch] gh/IvanKobzarev/26/base -> origin/gh/IvanKobzarev/26/base 2022-03-17T19:04:58.0349399Z * [new branch] gh/IvanKobzarev/26/head -> origin/gh/IvanKobzarev/26/head 2022-03-17T19:04:58.0350507Z * [new branch] gh/IvanKobzarev/26/orig -> origin/gh/IvanKobzarev/26/orig 2022-03-17T19:04:58.0351991Z * [new branch] gh/IvanKobzarev/29/base -> origin/gh/IvanKobzarev/29/base 2022-03-17T19:04:58.0353147Z * [new branch] gh/IvanKobzarev/29/head -> origin/gh/IvanKobzarev/29/head 2022-03-17T19:04:58.0354177Z * [new branch] gh/IvanKobzarev/29/orig -> origin/gh/IvanKobzarev/29/orig 2022-03-17T19:04:58.0355836Z * [new branch] gh/IvanKobzarev/44/base -> origin/gh/IvanKobzarev/44/base 2022-03-17T19:04:58.0357215Z * [new branch] gh/IvanKobzarev/44/head -> origin/gh/IvanKobzarev/44/head 2022-03-17T19:04:58.0358490Z * [new branch] gh/IvanKobzarev/44/orig -> origin/gh/IvanKobzarev/44/orig 2022-03-17T19:04:58.0360250Z * [new branch] gh/IvanKobzarev/47/base -> origin/gh/IvanKobzarev/47/base 2022-03-17T19:04:58.0361568Z * [new branch] gh/IvanKobzarev/47/head -> origin/gh/IvanKobzarev/47/head 2022-03-17T19:04:58.0363099Z * [new branch] gh/IvanKobzarev/47/orig -> origin/gh/IvanKobzarev/47/orig 2022-03-17T19:04:58.0364864Z * [new branch] gh/IvanKobzarev/60/base -> origin/gh/IvanKobzarev/60/base 2022-03-17T19:04:58.0366186Z * [new branch] gh/IvanKobzarev/60/head -> origin/gh/IvanKobzarev/60/head 2022-03-17T19:04:58.0367461Z * [new branch] gh/IvanKobzarev/60/orig -> origin/gh/IvanKobzarev/60/orig 2022-03-17T19:04:58.0369231Z * [new branch] gh/IvanKobzarev/61/base -> origin/gh/IvanKobzarev/61/base 2022-03-17T19:04:58.0370603Z * [new branch] gh/IvanKobzarev/61/head -> origin/gh/IvanKobzarev/61/head 2022-03-17T19:04:58.0371891Z * [new branch] gh/IvanKobzarev/61/orig -> origin/gh/IvanKobzarev/61/orig 2022-03-17T19:04:58.0373419Z * [new branch] gh/IvanKobzarev/62/base -> origin/gh/IvanKobzarev/62/base 2022-03-17T19:04:58.0374674Z * [new branch] gh/IvanKobzarev/62/head -> origin/gh/IvanKobzarev/62/head 2022-03-17T19:04:58.0375996Z * [new branch] gh/IvanKobzarev/62/orig -> origin/gh/IvanKobzarev/62/orig 2022-03-17T19:04:58.0377533Z * [new branch] gh/IvanKobzarev/63/base -> origin/gh/IvanKobzarev/63/base 2022-03-17T19:04:58.0379143Z * [new branch] gh/IvanKobzarev/63/head -> origin/gh/IvanKobzarev/63/head 2022-03-17T19:04:58.0380384Z * [new branch] gh/IvanKobzarev/63/orig -> origin/gh/IvanKobzarev/63/orig 2022-03-17T19:04:58.0382097Z * [new branch] gh/IvanKobzarev/67/base -> origin/gh/IvanKobzarev/67/base 2022-03-17T19:04:58.0383443Z * [new branch] gh/IvanKobzarev/67/head -> origin/gh/IvanKobzarev/67/head 2022-03-17T19:04:58.0384750Z * [new branch] gh/IvanKobzarev/67/orig -> origin/gh/IvanKobzarev/67/orig 2022-03-17T19:04:58.0386433Z * [new branch] gh/IvanKobzarev/68/base -> origin/gh/IvanKobzarev/68/base 2022-03-17T19:04:58.0387735Z * [new branch] gh/IvanKobzarev/68/head -> origin/gh/IvanKobzarev/68/head 2022-03-17T19:04:58.0389080Z * [new branch] gh/IvanKobzarev/68/orig -> origin/gh/IvanKobzarev/68/orig 2022-03-17T19:04:58.0390722Z * [new branch] gh/IvanKobzarev/74/base -> origin/gh/IvanKobzarev/74/base 2022-03-17T19:04:58.0392210Z * [new branch] gh/IvanKobzarev/74/head -> origin/gh/IvanKobzarev/74/head 2022-03-17T19:04:58.0479207Z * [new branch] gh/IvanKobzarev/74/orig -> origin/gh/IvanKobzarev/74/orig 2022-03-17T19:04:58.0481641Z * [new branch] gh/IvanKobzarev/83/base -> origin/gh/IvanKobzarev/83/base 2022-03-17T19:04:58.0483462Z * [new branch] gh/IvanKobzarev/83/head -> origin/gh/IvanKobzarev/83/head 2022-03-17T19:04:58.0484821Z * [new branch] gh/IvanKobzarev/83/orig -> origin/gh/IvanKobzarev/83/orig 2022-03-17T19:04:58.0486154Z * [new branch] gh/IvanKobzarev/85/base -> origin/gh/IvanKobzarev/85/base 2022-03-17T19:04:58.0487821Z * [new branch] gh/IvanKobzarev/85/head -> origin/gh/IvanKobzarev/85/head 2022-03-17T19:04:58.0490221Z * [new branch] gh/IvanKobzarev/85/orig -> origin/gh/IvanKobzarev/85/orig 2022-03-17T19:04:58.0491912Z * [new branch] gh/Lezcano/16/base -> origin/gh/Lezcano/16/base 2022-03-17T19:04:58.0493521Z * [new branch] gh/Lezcano/16/head -> origin/gh/Lezcano/16/head 2022-03-17T19:04:58.0495112Z * [new branch] gh/Lezcano/16/orig -> origin/gh/Lezcano/16/orig 2022-03-17T19:04:58.0496634Z * [new branch] gh/Lezcano/25/base -> origin/gh/Lezcano/25/base 2022-03-17T19:04:58.0498154Z * [new branch] gh/Lezcano/25/head -> origin/gh/Lezcano/25/head 2022-03-17T19:04:58.0499690Z * [new branch] gh/Lezcano/25/orig -> origin/gh/Lezcano/25/orig 2022-03-17T19:04:58.0501194Z * [new branch] gh/Lezcano/28/base -> origin/gh/Lezcano/28/base 2022-03-17T19:04:58.0502729Z * [new branch] gh/Lezcano/28/head -> origin/gh/Lezcano/28/head 2022-03-17T19:04:58.0504234Z * [new branch] gh/Lezcano/28/orig -> origin/gh/Lezcano/28/orig 2022-03-17T19:04:58.0506089Z * [new branch] gh/Lezcano/46/base -> origin/gh/Lezcano/46/base 2022-03-17T19:04:58.0507594Z * [new branch] gh/Lezcano/46/head -> origin/gh/Lezcano/46/head 2022-03-17T19:04:58.0509118Z * [new branch] gh/Lezcano/46/orig -> origin/gh/Lezcano/46/orig 2022-03-17T19:04:58.0510612Z * [new branch] gh/Lezcano/47/base -> origin/gh/Lezcano/47/base 2022-03-17T19:04:58.0512101Z * [new branch] gh/Lezcano/47/head -> origin/gh/Lezcano/47/head 2022-03-17T19:04:58.0513614Z * [new branch] gh/Lezcano/47/orig -> origin/gh/Lezcano/47/orig 2022-03-17T19:04:58.0515140Z * [new branch] gh/Lezcano/48/base -> origin/gh/Lezcano/48/base 2022-03-17T19:04:58.0516625Z * [new branch] gh/Lezcano/48/head -> origin/gh/Lezcano/48/head 2022-03-17T19:04:58.0518167Z * [new branch] gh/Lezcano/48/orig -> origin/gh/Lezcano/48/orig 2022-03-17T19:04:58.0519670Z * [new branch] gh/Lezcano/49/base -> origin/gh/Lezcano/49/base 2022-03-17T19:04:58.0521186Z * [new branch] gh/Lezcano/49/head -> origin/gh/Lezcano/49/head 2022-03-17T19:04:58.0523111Z * [new branch] gh/Lezcano/49/orig -> origin/gh/Lezcano/49/orig 2022-03-17T19:04:58.0524632Z * [new branch] gh/Lezcano/50/base -> origin/gh/Lezcano/50/base 2022-03-17T19:04:58.0526118Z * [new branch] gh/Lezcano/50/head -> origin/gh/Lezcano/50/head 2022-03-17T19:04:58.0527593Z * [new branch] gh/Lezcano/50/orig -> origin/gh/Lezcano/50/orig 2022-03-17T19:04:58.0529115Z * [new branch] gh/Lezcano/51/base -> origin/gh/Lezcano/51/base 2022-03-17T19:04:58.0530635Z * [new branch] gh/Lezcano/51/head -> origin/gh/Lezcano/51/head 2022-03-17T19:04:58.0532155Z * [new branch] gh/Lezcano/51/orig -> origin/gh/Lezcano/51/orig 2022-03-17T19:04:58.0533677Z * [new branch] gh/Lezcano/52/base -> origin/gh/Lezcano/52/base 2022-03-17T19:04:58.0535184Z * [new branch] gh/Lezcano/52/head -> origin/gh/Lezcano/52/head 2022-03-17T19:04:58.0536649Z * [new branch] gh/Lezcano/52/orig -> origin/gh/Lezcano/52/orig 2022-03-17T19:04:58.0537356Z * [new branch] gh/Lezcano/53/base -> origin/gh/Lezcano/53/base 2022-03-17T19:04:58.0537911Z * [new branch] gh/Lezcano/53/head -> origin/gh/Lezcano/53/head 2022-03-17T19:04:58.0538450Z * [new branch] gh/Lezcano/53/orig -> origin/gh/Lezcano/53/orig 2022-03-17T19:04:58.0539039Z * [new branch] gh/Lezcano/54/base -> origin/gh/Lezcano/54/base 2022-03-17T19:04:58.0539602Z * [new branch] gh/Lezcano/54/head -> origin/gh/Lezcano/54/head 2022-03-17T19:04:58.0540277Z * [new branch] gh/Lezcano/54/orig -> origin/gh/Lezcano/54/orig 2022-03-17T19:04:58.0540837Z * [new branch] gh/Lilyjjo/16/base -> origin/gh/Lilyjjo/16/base 2022-03-17T19:04:58.0541284Z * [new branch] gh/Lilyjjo/16/orig -> origin/gh/Lilyjjo/16/orig 2022-03-17T19:04:58.0541739Z * [new branch] gh/Lilyjjo/19/base -> origin/gh/Lilyjjo/19/base 2022-03-17T19:04:58.0542322Z * [new branch] gh/Lilyjjo/19/orig -> origin/gh/Lilyjjo/19/orig 2022-03-17T19:04:58.0542869Z * [new branch] gh/Lilyjjo/2/base -> origin/gh/Lilyjjo/2/base 2022-03-17T19:04:58.0543371Z * [new branch] gh/Lilyjjo/2/orig -> origin/gh/Lilyjjo/2/orig 2022-03-17T19:04:58.0543936Z * [new branch] gh/Lilyjjo/20/base -> origin/gh/Lilyjjo/20/base 2022-03-17T19:04:58.0544468Z * [new branch] gh/Lilyjjo/20/orig -> origin/gh/Lilyjjo/20/orig 2022-03-17T19:04:58.0545047Z * [new branch] gh/Lilyjjo/21/base -> origin/gh/Lilyjjo/21/base 2022-03-17T19:04:58.0545707Z * [new branch] gh/Lilyjjo/21/orig -> origin/gh/Lilyjjo/21/orig 2022-03-17T19:04:58.0546230Z * [new branch] gh/Lilyjjo/22/base -> origin/gh/Lilyjjo/22/base 2022-03-17T19:04:58.0546789Z * [new branch] gh/Lilyjjo/22/orig -> origin/gh/Lilyjjo/22/orig 2022-03-17T19:04:58.0547323Z * [new branch] gh/Lilyjjo/72/base -> origin/gh/Lilyjjo/72/base 2022-03-17T19:04:58.0547916Z * [new branch] gh/Lilyjjo/72/head -> origin/gh/Lilyjjo/72/head 2022-03-17T19:04:58.0548464Z * [new branch] gh/Lilyjjo/73/base -> origin/gh/Lilyjjo/73/base 2022-03-17T19:04:58.0549005Z * [new branch] gh/Lilyjjo/73/orig -> origin/gh/Lilyjjo/73/orig 2022-03-17T19:04:58.0549584Z * [new branch] gh/NicolasHug/1/base -> origin/gh/NicolasHug/1/base 2022-03-17T19:04:58.0550128Z * [new branch] gh/NicolasHug/1/head -> origin/gh/NicolasHug/1/head 2022-03-17T19:04:58.0550711Z * [new branch] gh/NicolasHug/1/orig -> origin/gh/NicolasHug/1/orig 2022-03-17T19:04:58.0551340Z * [new branch] gh/NicolasHug/2/base -> origin/gh/NicolasHug/2/base 2022-03-17T19:04:58.0551917Z * [new branch] gh/NicolasHug/2/head -> origin/gh/NicolasHug/2/head 2022-03-17T19:04:58.0552498Z * [new branch] gh/NicolasHug/2/orig -> origin/gh/NicolasHug/2/orig 2022-03-17T19:04:58.0553059Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2022-03-17T19:04:58.0553636Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2022-03-17T19:04:58.0554173Z * [new branch] gh/PaliC/105/base -> origin/gh/PaliC/105/base 2022-03-17T19:04:58.0554669Z * [new branch] gh/PaliC/105/head -> origin/gh/PaliC/105/head 2022-03-17T19:04:58.0555247Z * [new branch] gh/PaliC/106/base -> origin/gh/PaliC/106/base 2022-03-17T19:04:58.0555794Z * [new branch] gh/PaliC/106/orig -> origin/gh/PaliC/106/orig 2022-03-17T19:04:58.0556358Z * [new branch] gh/PaliC/110/base -> origin/gh/PaliC/110/base 2022-03-17T19:04:58.0556914Z * [new branch] gh/PaliC/110/head -> origin/gh/PaliC/110/head 2022-03-17T19:04:58.0557441Z * [new branch] gh/PaliC/111/base -> origin/gh/PaliC/111/base 2022-03-17T19:04:58.0557998Z * [new branch] gh/PaliC/111/orig -> origin/gh/PaliC/111/orig 2022-03-17T19:04:58.0558532Z * [new branch] gh/PaliC/112/base -> origin/gh/PaliC/112/base 2022-03-17T19:04:58.0559083Z * [new branch] gh/PaliC/112/orig -> origin/gh/PaliC/112/orig 2022-03-17T19:04:58.0559636Z * [new branch] gh/PaliC/113/base -> origin/gh/PaliC/113/base 2022-03-17T19:04:58.0560281Z * [new branch] gh/PaliC/113/orig -> origin/gh/PaliC/113/orig 2022-03-17T19:04:58.0560859Z * [new branch] gh/PaliC/114/base -> origin/gh/PaliC/114/base 2022-03-17T19:04:58.0561368Z * [new branch] gh/PaliC/114/head -> origin/gh/PaliC/114/head 2022-03-17T19:04:58.0562049Z * [new branch] gh/PaliC/115/base -> origin/gh/PaliC/115/base 2022-03-17T19:04:58.0562610Z * [new branch] gh/PaliC/115/orig -> origin/gh/PaliC/115/orig 2022-03-17T19:04:58.0563119Z * [new branch] gh/PaliC/116/base -> origin/gh/PaliC/116/base 2022-03-17T19:04:58.0563602Z * [new branch] gh/PaliC/116/orig -> origin/gh/PaliC/116/orig 2022-03-17T19:04:58.0564154Z * [new branch] gh/PaliC/117/base -> origin/gh/PaliC/117/base 2022-03-17T19:04:58.0564803Z * [new branch] gh/PaliC/117/orig -> origin/gh/PaliC/117/orig 2022-03-17T19:04:58.0565521Z * [new branch] gh/PaliC/120/base -> origin/gh/PaliC/120/base 2022-03-17T19:04:58.0566366Z * [new branch] gh/PaliC/120/head -> origin/gh/PaliC/120/head 2022-03-17T19:04:58.0567096Z * [new branch] gh/PaliC/120/orig -> origin/gh/PaliC/120/orig 2022-03-17T19:04:58.0567740Z * [new branch] gh/PaliC/121/base -> origin/gh/PaliC/121/base 2022-03-17T19:04:58.0568335Z * [new branch] gh/PaliC/121/head -> origin/gh/PaliC/121/head 2022-03-17T19:04:58.0569045Z * [new branch] gh/PaliC/121/orig -> origin/gh/PaliC/121/orig 2022-03-17T19:04:58.0569741Z * [new branch] gh/PaliC/122/base -> origin/gh/PaliC/122/base 2022-03-17T19:04:58.0570425Z * [new branch] gh/PaliC/122/head -> origin/gh/PaliC/122/head 2022-03-17T19:04:58.0571089Z * [new branch] gh/PaliC/122/orig -> origin/gh/PaliC/122/orig 2022-03-17T19:04:58.0571776Z * [new branch] gh/PaliC/123/base -> origin/gh/PaliC/123/base 2022-03-17T19:04:58.0572427Z * [new branch] gh/PaliC/123/head -> origin/gh/PaliC/123/head 2022-03-17T19:04:58.0573141Z * [new branch] gh/PaliC/123/orig -> origin/gh/PaliC/123/orig 2022-03-17T19:04:58.0573809Z * [new branch] gh/PaliC/124/base -> origin/gh/PaliC/124/base 2022-03-17T19:04:58.0574541Z * [new branch] gh/PaliC/124/head -> origin/gh/PaliC/124/head 2022-03-17T19:04:58.0575204Z * [new branch] gh/PaliC/124/orig -> origin/gh/PaliC/124/orig 2022-03-17T19:04:58.0575788Z * [new branch] gh/PaliC/126/base -> origin/gh/PaliC/126/base 2022-03-17T19:04:58.0576360Z * [new branch] gh/PaliC/126/head -> origin/gh/PaliC/126/head 2022-03-17T19:04:58.0576926Z * [new branch] gh/PaliC/126/orig -> origin/gh/PaliC/126/orig 2022-03-17T19:04:58.0577391Z * [new branch] gh/PaliC/127/base -> origin/gh/PaliC/127/base 2022-03-17T19:04:58.0577872Z * [new branch] gh/PaliC/127/head -> origin/gh/PaliC/127/head 2022-03-17T19:04:58.0578358Z * [new branch] gh/PaliC/127/orig -> origin/gh/PaliC/127/orig 2022-03-17T19:04:58.0578824Z * [new branch] gh/PaliC/128/base -> origin/gh/PaliC/128/base 2022-03-17T19:04:58.0579306Z * [new branch] gh/PaliC/128/head -> origin/gh/PaliC/128/head 2022-03-17T19:04:58.0579776Z * [new branch] gh/PaliC/128/orig -> origin/gh/PaliC/128/orig 2022-03-17T19:04:58.0580255Z * [new branch] gh/PaliC/129/base -> origin/gh/PaliC/129/base 2022-03-17T19:04:58.0580732Z * [new branch] gh/PaliC/129/head -> origin/gh/PaliC/129/head 2022-03-17T19:04:58.0581205Z * [new branch] gh/PaliC/129/orig -> origin/gh/PaliC/129/orig 2022-03-17T19:04:58.0581681Z * [new branch] gh/PaliC/130/base -> origin/gh/PaliC/130/base 2022-03-17T19:04:58.0582255Z * [new branch] gh/PaliC/130/head -> origin/gh/PaliC/130/head 2022-03-17T19:04:58.0593673Z * [new branch] gh/PaliC/130/orig -> origin/gh/PaliC/130/orig 2022-03-17T19:04:58.0594285Z * [new branch] gh/PaliC/131/base -> origin/gh/PaliC/131/base 2022-03-17T19:04:58.0594955Z * [new branch] gh/PaliC/131/head -> origin/gh/PaliC/131/head 2022-03-17T19:04:58.0595653Z * [new branch] gh/PaliC/131/orig -> origin/gh/PaliC/131/orig 2022-03-17T19:04:58.0596327Z * [new branch] gh/PaliC/132/base -> origin/gh/PaliC/132/base 2022-03-17T19:04:58.0596995Z * [new branch] gh/PaliC/132/head -> origin/gh/PaliC/132/head 2022-03-17T19:04:58.0597668Z * [new branch] gh/PaliC/132/orig -> origin/gh/PaliC/132/orig 2022-03-17T19:04:58.0598308Z * [new branch] gh/PaliC/133/base -> origin/gh/PaliC/133/base 2022-03-17T19:04:58.0599003Z * [new branch] gh/PaliC/133/head -> origin/gh/PaliC/133/head 2022-03-17T19:04:58.0599853Z * [new branch] gh/PaliC/133/orig -> origin/gh/PaliC/133/orig 2022-03-17T19:04:58.0600519Z * [new branch] gh/PaliC/134/base -> origin/gh/PaliC/134/base 2022-03-17T19:04:58.0601214Z * [new branch] gh/PaliC/134/head -> origin/gh/PaliC/134/head 2022-03-17T19:04:58.0601955Z * [new branch] gh/PaliC/134/orig -> origin/gh/PaliC/134/orig 2022-03-17T19:04:58.0602563Z * [new branch] gh/PaliC/135/base -> origin/gh/PaliC/135/base 2022-03-17T19:04:58.0603226Z * [new branch] gh/PaliC/135/head -> origin/gh/PaliC/135/head 2022-03-17T19:04:58.0603916Z * [new branch] gh/PaliC/135/orig -> origin/gh/PaliC/135/orig 2022-03-17T19:04:58.0604604Z * [new branch] gh/PaliC/136/base -> origin/gh/PaliC/136/base 2022-03-17T19:04:58.0605293Z * [new branch] gh/PaliC/136/head -> origin/gh/PaliC/136/head 2022-03-17T19:04:58.0605967Z * [new branch] gh/PaliC/136/orig -> origin/gh/PaliC/136/orig 2022-03-17T19:04:58.0606652Z * [new branch] gh/PaliC/137/base -> origin/gh/PaliC/137/base 2022-03-17T19:04:58.0607270Z * [new branch] gh/PaliC/137/head -> origin/gh/PaliC/137/head 2022-03-17T19:04:58.0607985Z * [new branch] gh/PaliC/137/orig -> origin/gh/PaliC/137/orig 2022-03-17T19:04:58.0608737Z * [new branch] gh/PaliC/138/base -> origin/gh/PaliC/138/base 2022-03-17T19:04:58.0609422Z * [new branch] gh/PaliC/138/head -> origin/gh/PaliC/138/head 2022-03-17T19:04:58.0610256Z * [new branch] gh/PaliC/138/orig -> origin/gh/PaliC/138/orig 2022-03-17T19:04:58.0612075Z * [new branch] gh/PaliC/139/base -> origin/gh/PaliC/139/base 2022-03-17T19:04:58.0613233Z * [new branch] gh/PaliC/139/head -> origin/gh/PaliC/139/head 2022-03-17T19:04:58.0614688Z * [new branch] gh/PaliC/139/orig -> origin/gh/PaliC/139/orig 2022-03-17T19:04:58.0616372Z * [new branch] gh/PaliC/140/base -> origin/gh/PaliC/140/base 2022-03-17T19:04:58.0617715Z * [new branch] gh/PaliC/140/head -> origin/gh/PaliC/140/head 2022-03-17T19:04:58.0618933Z * [new branch] gh/PaliC/140/orig -> origin/gh/PaliC/140/orig 2022-03-17T19:04:58.0620669Z * [new branch] gh/PaliC/142/base -> origin/gh/PaliC/142/base 2022-03-17T19:04:58.0622081Z * [new branch] gh/PaliC/142/head -> origin/gh/PaliC/142/head 2022-03-17T19:04:58.0623403Z * [new branch] gh/PaliC/142/orig -> origin/gh/PaliC/142/orig 2022-03-17T19:04:58.0625151Z * [new branch] gh/PaliC/143/base -> origin/gh/PaliC/143/base 2022-03-17T19:04:58.0626555Z * [new branch] gh/PaliC/143/head -> origin/gh/PaliC/143/head 2022-03-17T19:04:58.0628075Z * [new branch] gh/PaliC/143/orig -> origin/gh/PaliC/143/orig 2022-03-17T19:04:58.0629670Z * [new branch] gh/PaliC/144/base -> origin/gh/PaliC/144/base 2022-03-17T19:04:58.0630961Z * [new branch] gh/PaliC/144/head -> origin/gh/PaliC/144/head 2022-03-17T19:04:58.0632782Z * [new branch] gh/PaliC/145/base -> origin/gh/PaliC/145/base 2022-03-17T19:04:58.0634107Z * [new branch] gh/PaliC/145/head -> origin/gh/PaliC/145/head 2022-03-17T19:04:58.0635469Z * [new branch] gh/PaliC/145/orig -> origin/gh/PaliC/145/orig 2022-03-17T19:04:58.0637250Z * [new branch] gh/PaliC/146/base -> origin/gh/PaliC/146/base 2022-03-17T19:04:58.0638625Z * [new branch] gh/PaliC/146/head -> origin/gh/PaliC/146/head 2022-03-17T19:04:58.0639994Z * [new branch] gh/PaliC/146/orig -> origin/gh/PaliC/146/orig 2022-03-17T19:04:58.0641984Z * [new branch] gh/PaliC/147/base -> origin/gh/PaliC/147/base 2022-03-17T19:04:58.0643542Z * [new branch] gh/PaliC/147/head -> origin/gh/PaliC/147/head 2022-03-17T19:04:58.0644875Z * [new branch] gh/PaliC/147/orig -> origin/gh/PaliC/147/orig 2022-03-17T19:04:58.0646679Z * [new branch] gh/PaliC/148/base -> origin/gh/PaliC/148/base 2022-03-17T19:04:58.0648166Z * [new branch] gh/PaliC/148/head -> origin/gh/PaliC/148/head 2022-03-17T19:04:58.0649412Z * [new branch] gh/PaliC/148/orig -> origin/gh/PaliC/148/orig 2022-03-17T19:04:58.0651154Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2022-03-17T19:04:58.0652372Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2022-03-17T19:04:58.0654142Z * [new branch] gh/PaliC/42/base -> origin/gh/PaliC/42/base 2022-03-17T19:04:58.0655379Z * [new branch] gh/PaliC/42/orig -> origin/gh/PaliC/42/orig 2022-03-17T19:04:58.0657213Z * [new branch] gh/PaliC/5/base -> origin/gh/PaliC/5/base 2022-03-17T19:04:58.0658466Z * [new branch] gh/PaliC/5/orig -> origin/gh/PaliC/5/orig 2022-03-17T19:04:58.0660239Z * [new branch] gh/PaliC/56/base -> origin/gh/PaliC/56/base 2022-03-17T19:04:58.0661606Z * [new branch] gh/PaliC/56/head -> origin/gh/PaliC/56/head 2022-03-17T19:04:58.0664002Z * [new branch] gh/PaliC/56/orig -> origin/gh/PaliC/56/orig 2022-03-17T19:04:58.0664598Z * [new branch] gh/PaliC/60/base -> origin/gh/PaliC/60/base 2022-03-17T19:04:58.0665614Z * [new branch] gh/PaliC/60/orig -> origin/gh/PaliC/60/orig 2022-03-17T19:04:58.0667506Z * [new branch] gh/PaliC/72/base -> origin/gh/PaliC/72/base 2022-03-17T19:04:58.0669197Z * [new branch] gh/PaliC/94/base -> origin/gh/PaliC/94/base 2022-03-17T19:04:58.0671065Z * [new branch] gh/SS-JIA/21/base -> origin/gh/SS-JIA/21/base 2022-03-17T19:04:58.0672453Z * [new branch] gh/SS-JIA/21/head -> origin/gh/SS-JIA/21/head 2022-03-17T19:04:58.0673751Z * [new branch] gh/SS-JIA/21/orig -> origin/gh/SS-JIA/21/orig 2022-03-17T19:04:58.0675450Z * [new branch] gh/SS-JIA/23/base -> origin/gh/SS-JIA/23/base 2022-03-17T19:04:58.0676725Z * [new branch] gh/SS-JIA/23/head -> origin/gh/SS-JIA/23/head 2022-03-17T19:04:58.0678026Z * [new branch] gh/SS-JIA/23/orig -> origin/gh/SS-JIA/23/orig 2022-03-17T19:04:58.0679800Z * [new branch] gh/SS-JIA/24/base -> origin/gh/SS-JIA/24/base 2022-03-17T19:04:58.0681191Z * [new branch] gh/SS-JIA/24/head -> origin/gh/SS-JIA/24/head 2022-03-17T19:04:58.0682927Z * [new branch] gh/SS-JIA/24/orig -> origin/gh/SS-JIA/24/orig 2022-03-17T19:04:58.0685356Z * [new branch] gh/SS-JIA/25/base -> origin/gh/SS-JIA/25/base 2022-03-17T19:04:58.0686602Z * [new branch] gh/SS-JIA/25/head -> origin/gh/SS-JIA/25/head 2022-03-17T19:04:58.0687933Z * [new branch] gh/SS-JIA/25/orig -> origin/gh/SS-JIA/25/orig 2022-03-17T19:04:58.0689581Z * [new branch] gh/SS-JIA/26/base -> origin/gh/SS-JIA/26/base 2022-03-17T19:04:58.0690882Z * [new branch] gh/SS-JIA/26/head -> origin/gh/SS-JIA/26/head 2022-03-17T19:04:58.0692148Z * [new branch] gh/SS-JIA/26/orig -> origin/gh/SS-JIA/26/orig 2022-03-17T19:04:58.0693741Z * [new branch] gh/SS-JIA/29/base -> origin/gh/SS-JIA/29/base 2022-03-17T19:04:58.0695107Z * [new branch] gh/SS-JIA/29/head -> origin/gh/SS-JIA/29/head 2022-03-17T19:04:58.0696440Z * [new branch] gh/SS-JIA/29/orig -> origin/gh/SS-JIA/29/orig 2022-03-17T19:04:58.0698194Z * [new branch] gh/SS-JIA/30/base -> origin/gh/SS-JIA/30/base 2022-03-17T19:04:58.0699494Z * [new branch] gh/SS-JIA/30/orig -> origin/gh/SS-JIA/30/orig 2022-03-17T19:04:58.0701173Z * [new branch] gh/SS-JIA/31/base -> origin/gh/SS-JIA/31/base 2022-03-17T19:04:58.0702462Z * [new branch] gh/SS-JIA/31/orig -> origin/gh/SS-JIA/31/orig 2022-03-17T19:04:58.0704078Z * [new branch] gh/SS-JIA/32/base -> origin/gh/SS-JIA/32/base 2022-03-17T19:04:58.0705391Z * [new branch] gh/SS-JIA/32/orig -> origin/gh/SS-JIA/32/orig 2022-03-17T19:04:58.0707031Z * [new branch] gh/SS-JIA/33/base -> origin/gh/SS-JIA/33/base 2022-03-17T19:04:58.0708397Z * [new branch] gh/SS-JIA/33/orig -> origin/gh/SS-JIA/33/orig 2022-03-17T19:04:58.0710074Z * [new branch] gh/SS-JIA/34/base -> origin/gh/SS-JIA/34/base 2022-03-17T19:04:58.0711434Z * [new branch] gh/SS-JIA/34/orig -> origin/gh/SS-JIA/34/orig 2022-03-17T19:04:58.0713042Z * [new branch] gh/SS-JIA/35/base -> origin/gh/SS-JIA/35/base 2022-03-17T19:04:58.0714363Z * [new branch] gh/SS-JIA/35/orig -> origin/gh/SS-JIA/35/orig 2022-03-17T19:04:58.0716005Z * [new branch] gh/SS-JIA/36/base -> origin/gh/SS-JIA/36/base 2022-03-17T19:04:58.0717390Z * [new branch] gh/SS-JIA/36/orig -> origin/gh/SS-JIA/36/orig 2022-03-17T19:04:58.0719180Z * [new branch] gh/SS-JIA/40/base -> origin/gh/SS-JIA/40/base 2022-03-17T19:04:58.0720418Z * [new branch] gh/SS-JIA/40/head -> origin/gh/SS-JIA/40/head 2022-03-17T19:04:58.0721783Z * [new branch] gh/SS-JIA/40/orig -> origin/gh/SS-JIA/40/orig 2022-03-17T19:04:58.0723599Z * [new branch] gh/SS-JIA/41/base -> origin/gh/SS-JIA/41/base 2022-03-17T19:04:58.0724873Z * [new branch] gh/SS-JIA/41/orig -> origin/gh/SS-JIA/41/orig 2022-03-17T19:04:58.0726346Z * [new branch] gh/SS-JIA/42/base -> origin/gh/SS-JIA/42/base 2022-03-17T19:04:58.0727805Z * [new branch] gh/SS-JIA/42/orig -> origin/gh/SS-JIA/42/orig 2022-03-17T19:04:58.0729353Z * [new branch] gh/SS-JIA/43/base -> origin/gh/SS-JIA/43/base 2022-03-17T19:04:58.0730549Z * [new branch] gh/SS-JIA/43/head -> origin/gh/SS-JIA/43/head 2022-03-17T19:04:58.0731815Z * [new branch] gh/SS-JIA/43/orig -> origin/gh/SS-JIA/43/orig 2022-03-17T19:04:58.0733850Z * [new branch] gh/SciPioneer/1/base -> origin/gh/SciPioneer/1/base 2022-03-17T19:04:58.0735148Z * [new branch] gh/SciPioneer/1/head -> origin/gh/SciPioneer/1/head 2022-03-17T19:04:58.0737148Z * [new branch] gh/SciPioneer/10/base -> origin/gh/SciPioneer/10/base 2022-03-17T19:04:58.0738595Z * [new branch] gh/SciPioneer/10/head -> origin/gh/SciPioneer/10/head 2022-03-17T19:04:58.0739974Z * [new branch] gh/SciPioneer/11/base -> origin/gh/SciPioneer/11/base 2022-03-17T19:04:58.0741142Z * [new branch] gh/SciPioneer/11/head -> origin/gh/SciPioneer/11/head 2022-03-17T19:04:58.0743127Z * [new branch] gh/SciPioneer/147/base -> origin/gh/SciPioneer/147/base 2022-03-17T19:04:58.0744220Z * [new branch] gh/SciPioneer/147/head -> origin/gh/SciPioneer/147/head 2022-03-17T19:04:58.0745647Z * [new branch] gh/SciPioneer/147/orig -> origin/gh/SciPioneer/147/orig 2022-03-17T19:04:58.0747445Z * [new branch] gh/SciPioneer/2/base -> origin/gh/SciPioneer/2/base 2022-03-17T19:04:58.0748713Z * [new branch] gh/SciPioneer/2/head -> origin/gh/SciPioneer/2/head 2022-03-17T19:04:58.0750819Z * [new branch] gh/SciPioneer/22/base -> origin/gh/SciPioneer/22/base 2022-03-17T19:04:58.0751869Z * [new branch] gh/SciPioneer/22/head -> origin/gh/SciPioneer/22/head 2022-03-17T19:04:58.0753460Z * [new branch] gh/SciPioneer/3/base -> origin/gh/SciPioneer/3/base 2022-03-17T19:04:58.0754629Z * [new branch] gh/SciPioneer/3/head -> origin/gh/SciPioneer/3/head 2022-03-17T19:04:58.0756750Z * [new branch] gh/SciPioneer/36/base -> origin/gh/SciPioneer/36/base 2022-03-17T19:04:58.0757789Z * [new branch] gh/SciPioneer/36/head -> origin/gh/SciPioneer/36/head 2022-03-17T19:04:58.0759230Z * [new branch] gh/SciPioneer/36/orig -> origin/gh/SciPioneer/36/orig 2022-03-17T19:04:58.0760758Z * [new branch] gh/SciPioneer/37/base -> origin/gh/SciPioneer/37/base 2022-03-17T19:04:58.0762367Z * [new branch] gh/SciPioneer/37/head -> origin/gh/SciPioneer/37/head 2022-03-17T19:04:58.0764105Z * [new branch] gh/SciPioneer/37/orig -> origin/gh/SciPioneer/37/orig 2022-03-17T19:04:58.0765648Z * [new branch] gh/SciPioneer/4/base -> origin/gh/SciPioneer/4/base 2022-03-17T19:04:58.0767508Z * [new branch] gh/SciPioneer/4/head -> origin/gh/SciPioneer/4/head 2022-03-17T19:04:58.0768866Z * [new branch] gh/SciPioneer/5/base -> origin/gh/SciPioneer/5/base 2022-03-17T19:04:58.0770278Z * [new branch] gh/SciPioneer/5/head -> origin/gh/SciPioneer/5/head 2022-03-17T19:04:58.0771587Z * [new branch] gh/SciPioneer/6/base -> origin/gh/SciPioneer/6/base 2022-03-17T19:04:58.0772838Z * [new branch] gh/SciPioneer/6/head -> origin/gh/SciPioneer/6/head 2022-03-17T19:04:58.0774065Z * [new branch] gh/SciPioneer/7/base -> origin/gh/SciPioneer/7/base 2022-03-17T19:04:58.0775272Z * [new branch] gh/SciPioneer/7/head -> origin/gh/SciPioneer/7/head 2022-03-17T19:04:58.0776831Z * [new branch] gh/SciPioneer/70/base -> origin/gh/SciPioneer/70/base 2022-03-17T19:04:58.0778185Z * [new branch] gh/SciPioneer/70/head -> origin/gh/SciPioneer/70/head 2022-03-17T19:04:58.0779438Z * [new branch] gh/SciPioneer/70/orig -> origin/gh/SciPioneer/70/orig 2022-03-17T19:04:58.0781425Z * [new branch] gh/SciPioneer/8/base -> origin/gh/SciPioneer/8/base 2022-03-17T19:04:58.0782707Z * [new branch] gh/SciPioneer/8/head -> origin/gh/SciPioneer/8/head 2022-03-17T19:04:58.0784257Z * [new branch] gh/SciPioneer/9/base -> origin/gh/SciPioneer/9/base 2022-03-17T19:04:58.0785534Z * [new branch] gh/SciPioneer/9/head -> origin/gh/SciPioneer/9/head 2022-03-17T19:04:58.0787460Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2022-03-17T19:04:58.0789339Z * [new branch] gh/SiddhantRanade/1/base -> origin/gh/SiddhantRanade/1/base 2022-03-17T19:04:58.0790942Z * [new branch] gh/SiddhantRanade/1/head -> origin/gh/SiddhantRanade/1/head 2022-03-17T19:04:58.0792427Z * [new branch] gh/SiddhantRanade/1/orig -> origin/gh/SiddhantRanade/1/orig 2022-03-17T19:04:58.0794408Z * [new branch] gh/VitalyFedyunin/10/base -> origin/gh/VitalyFedyunin/10/base 2022-03-17T19:04:58.0795748Z * [new branch] gh/VitalyFedyunin/10/orig -> origin/gh/VitalyFedyunin/10/orig 2022-03-17T19:04:58.0797348Z * [new branch] gh/VitalyFedyunin/113/base -> origin/gh/VitalyFedyunin/113/base 2022-03-17T19:04:58.0798694Z * [new branch] gh/VitalyFedyunin/113/orig -> origin/gh/VitalyFedyunin/113/orig 2022-03-17T19:04:58.0800373Z * [new branch] gh/VitalyFedyunin/149/base -> origin/gh/VitalyFedyunin/149/base 2022-03-17T19:04:58.0801744Z * [new branch] gh/VitalyFedyunin/149/head -> origin/gh/VitalyFedyunin/149/head 2022-03-17T19:04:58.0803317Z * [new branch] gh/VitalyFedyunin/149/orig -> origin/gh/VitalyFedyunin/149/orig 2022-03-17T19:04:58.0805105Z * [new branch] gh/VitalyFedyunin/170/base -> origin/gh/VitalyFedyunin/170/base 2022-03-17T19:04:58.0806428Z * [new branch] gh/VitalyFedyunin/170/head -> origin/gh/VitalyFedyunin/170/head 2022-03-17T19:04:58.0807709Z * [new branch] gh/VitalyFedyunin/170/orig -> origin/gh/VitalyFedyunin/170/orig 2022-03-17T19:04:58.0809509Z * [new branch] gh/VitalyFedyunin/175/base -> origin/gh/VitalyFedyunin/175/base 2022-03-17T19:04:58.0810741Z * [new branch] gh/VitalyFedyunin/175/head -> origin/gh/VitalyFedyunin/175/head 2022-03-17T19:04:58.0812058Z * [new branch] gh/VitalyFedyunin/175/orig -> origin/gh/VitalyFedyunin/175/orig 2022-03-17T19:04:58.0813775Z * [new branch] gh/VitalyFedyunin/177/base -> origin/gh/VitalyFedyunin/177/base 2022-03-17T19:04:58.0815066Z * [new branch] gh/VitalyFedyunin/177/head -> origin/gh/VitalyFedyunin/177/head 2022-03-17T19:04:58.0816401Z * [new branch] gh/VitalyFedyunin/177/orig -> origin/gh/VitalyFedyunin/177/orig 2022-03-17T19:04:58.0817809Z * [new branch] gh/VitalyFedyunin/20/base -> origin/gh/VitalyFedyunin/20/base 2022-03-17T19:04:58.0819258Z * [new branch] gh/VitalyFedyunin/20/orig -> origin/gh/VitalyFedyunin/20/orig 2022-03-17T19:04:58.0821011Z * [new branch] gh/VitalyFedyunin/46/orig -> origin/gh/VitalyFedyunin/46/orig 2022-03-17T19:04:58.0822678Z * [new branch] gh/VitalyFedyunin/47/orig -> origin/gh/VitalyFedyunin/47/orig 2022-03-17T19:04:58.0824302Z * [new branch] gh/VitalyFedyunin/48/orig -> origin/gh/VitalyFedyunin/48/orig 2022-03-17T19:04:58.0825930Z * [new branch] gh/VitalyFedyunin/49/orig -> origin/gh/VitalyFedyunin/49/orig 2022-03-17T19:04:58.0827613Z * [new branch] gh/VitalyFedyunin/50/orig -> origin/gh/VitalyFedyunin/50/orig 2022-03-17T19:04:58.0829206Z * [new branch] gh/VitalyFedyunin/51/orig -> origin/gh/VitalyFedyunin/51/orig 2022-03-17T19:04:58.0830870Z * [new branch] gh/VitalyFedyunin/52/orig -> origin/gh/VitalyFedyunin/52/orig 2022-03-17T19:04:58.0832484Z * [new branch] gh/VitalyFedyunin/56/orig -> origin/gh/VitalyFedyunin/56/orig 2022-03-17T19:04:58.0834138Z * [new branch] gh/VitalyFedyunin/57/orig -> origin/gh/VitalyFedyunin/57/orig 2022-03-17T19:04:58.0835785Z * [new branch] gh/VitalyFedyunin/58/orig -> origin/gh/VitalyFedyunin/58/orig 2022-03-17T19:04:58.0837422Z * [new branch] gh/VitalyFedyunin/59/orig -> origin/gh/VitalyFedyunin/59/orig 2022-03-17T19:04:58.0839159Z * [new branch] gh/VitalyFedyunin/60/orig -> origin/gh/VitalyFedyunin/60/orig 2022-03-17T19:04:58.0840755Z * [new branch] gh/VitalyFedyunin/61/orig -> origin/gh/VitalyFedyunin/61/orig 2022-03-17T19:04:58.0842806Z * [new branch] gh/VitalyFedyunin/62/orig -> origin/gh/VitalyFedyunin/62/orig 2022-03-17T19:04:58.0845016Z * [new branch] gh/VitalyFedyunin/64/base -> origin/gh/VitalyFedyunin/64/base 2022-03-17T19:04:58.0846554Z * [new branch] gh/VitalyFedyunin/75/orig -> origin/gh/VitalyFedyunin/75/orig 2022-03-17T19:04:58.0848160Z * [new branch] gh/VitalyFedyunin/77/orig -> origin/gh/VitalyFedyunin/77/orig 2022-03-17T19:04:58.0849696Z * [new branch] gh/VitalyFedyunin/78/orig -> origin/gh/VitalyFedyunin/78/orig 2022-03-17T19:04:58.0851724Z * [new branch] gh/XiaobingSuper/9/base -> origin/gh/XiaobingSuper/9/base 2022-03-17T19:04:58.0853015Z * [new branch] gh/XiaobingSuper/9/orig -> origin/gh/XiaobingSuper/9/orig 2022-03-17T19:04:58.0854960Z * [new branch] gh/YazhiGao/1/base -> origin/gh/YazhiGao/1/base 2022-03-17T19:04:58.0856303Z * [new branch] gh/YazhiGao/1/orig -> origin/gh/YazhiGao/1/orig 2022-03-17T19:04:58.0858163Z * [new branch] gh/YesterdayDX/1/base -> origin/gh/YesterdayDX/1/base 2022-03-17T19:04:58.0859543Z * [new branch] gh/YesterdayDX/1/head -> origin/gh/YesterdayDX/1/head 2022-03-17T19:04:58.0861133Z * [new branch] gh/YesterdayDX/2/base -> origin/gh/YesterdayDX/2/base 2022-03-17T19:04:58.0862343Z * [new branch] gh/YesterdayDX/2/head -> origin/gh/YesterdayDX/2/head 2022-03-17T19:04:58.0863871Z * [new branch] gh/YesterdayDX/3/base -> origin/gh/YesterdayDX/3/base 2022-03-17T19:04:58.0865140Z * [new branch] gh/YesterdayDX/3/head -> origin/gh/YesterdayDX/3/head 2022-03-17T19:04:58.0867009Z * [new branch] gh/ZolotukhinM/1/base -> origin/gh/ZolotukhinM/1/base 2022-03-17T19:04:58.0868369Z * [new branch] gh/ZolotukhinM/1/head -> origin/gh/ZolotukhinM/1/head 2022-03-17T19:04:58.0869707Z * [new branch] gh/ZolotukhinM/1/orig -> origin/gh/ZolotukhinM/1/orig 2022-03-17T19:04:58.0871550Z * [new branch] gh/ZolotukhinM/192/base -> origin/gh/ZolotukhinM/192/base 2022-03-17T19:04:58.0872929Z * [new branch] gh/ZolotukhinM/192/head -> origin/gh/ZolotukhinM/192/head 2022-03-17T19:04:58.0874653Z * [new branch] gh/ZolotukhinM/22/base -> origin/gh/ZolotukhinM/22/base 2022-03-17T19:04:58.0876312Z * [new branch] gh/ZolotukhinM/242/head -> origin/gh/ZolotukhinM/242/head 2022-03-17T19:04:58.0877770Z * [new branch] gh/ZolotukhinM/242/orig -> origin/gh/ZolotukhinM/242/orig 2022-03-17T19:04:58.0879610Z * [new branch] gh/ZolotukhinM/354/base -> origin/gh/ZolotukhinM/354/base 2022-03-17T19:04:58.0880900Z * [new branch] gh/ZolotukhinM/354/head -> origin/gh/ZolotukhinM/354/head 2022-03-17T19:04:58.0882465Z * [new branch] gh/ZolotukhinM/354/orig -> origin/gh/ZolotukhinM/354/orig 2022-03-17T19:04:58.0884389Z * [new branch] gh/ZolotukhinM/355/base -> origin/gh/ZolotukhinM/355/base 2022-03-17T19:04:58.0885755Z * [new branch] gh/ZolotukhinM/355/head -> origin/gh/ZolotukhinM/355/head 2022-03-17T19:04:58.0887006Z * [new branch] gh/ZolotukhinM/355/orig -> origin/gh/ZolotukhinM/355/orig 2022-03-17T19:04:58.0888826Z * [new branch] gh/ZolotukhinM/404/base -> origin/gh/ZolotukhinM/404/base 2022-03-17T19:04:58.0890183Z * [new branch] gh/ZolotukhinM/404/head -> origin/gh/ZolotukhinM/404/head 2022-03-17T19:04:58.0891454Z * [new branch] gh/ZolotukhinM/404/orig -> origin/gh/ZolotukhinM/404/orig 2022-03-17T19:04:58.0892915Z * [new branch] gh/ZolotukhinM/405/base -> origin/gh/ZolotukhinM/405/base 2022-03-17T19:04:58.0894295Z * [new branch] gh/ZolotukhinM/405/head -> origin/gh/ZolotukhinM/405/head 2022-03-17T19:04:58.0895699Z * [new branch] gh/ZolotukhinM/405/orig -> origin/gh/ZolotukhinM/405/orig 2022-03-17T19:04:58.0897420Z * [new branch] gh/ZolotukhinM/443/base -> origin/gh/ZolotukhinM/443/base 2022-03-17T19:04:58.0898867Z * [new branch] gh/ZolotukhinM/443/head -> origin/gh/ZolotukhinM/443/head 2022-03-17T19:04:58.0899923Z * [new branch] gh/ZolotukhinM/443/orig -> origin/gh/ZolotukhinM/443/orig 2022-03-17T19:04:58.0901650Z * [new branch] gh/ZolotukhinM/444/base -> origin/gh/ZolotukhinM/444/base 2022-03-17T19:04:58.0903041Z * [new branch] gh/ZolotukhinM/444/head -> origin/gh/ZolotukhinM/444/head 2022-03-17T19:04:58.0904365Z * [new branch] gh/ZolotukhinM/444/orig -> origin/gh/ZolotukhinM/444/orig 2022-03-17T19:04:58.0906079Z * [new branch] gh/ZolotukhinM/448/base -> origin/gh/ZolotukhinM/448/base 2022-03-17T19:04:58.0907346Z * [new branch] gh/ZolotukhinM/448/head -> origin/gh/ZolotukhinM/448/head 2022-03-17T19:04:58.0908711Z * [new branch] gh/ZolotukhinM/448/orig -> origin/gh/ZolotukhinM/448/orig 2022-03-17T19:04:58.0910399Z * [new branch] gh/ZolotukhinM/449/base -> origin/gh/ZolotukhinM/449/base 2022-03-17T19:04:58.0911703Z * [new branch] gh/ZolotukhinM/449/head -> origin/gh/ZolotukhinM/449/head 2022-03-17T19:04:58.0912986Z * [new branch] gh/ZolotukhinM/449/orig -> origin/gh/ZolotukhinM/449/orig 2022-03-17T19:04:58.0914680Z * [new branch] gh/ZolotukhinM/454/base -> origin/gh/ZolotukhinM/454/base 2022-03-17T19:04:58.0916028Z * [new branch] gh/ZolotukhinM/454/head -> origin/gh/ZolotukhinM/454/head 2022-03-17T19:04:58.0917333Z * [new branch] gh/ZolotukhinM/454/orig -> origin/gh/ZolotukhinM/454/orig 2022-03-17T19:04:58.0919140Z * [new branch] gh/ZolotukhinM/464/base -> origin/gh/ZolotukhinM/464/base 2022-03-17T19:04:58.0920544Z * [new branch] gh/ZolotukhinM/464/head -> origin/gh/ZolotukhinM/464/head 2022-03-17T19:04:58.0921980Z * [new branch] gh/ZolotukhinM/464/orig -> origin/gh/ZolotukhinM/464/orig 2022-03-17T19:04:58.0923762Z * [new branch] gh/ZolotukhinM/468/base -> origin/gh/ZolotukhinM/468/base 2022-03-17T19:04:58.0925036Z * [new branch] gh/ZolotukhinM/468/head -> origin/gh/ZolotukhinM/468/head 2022-03-17T19:04:58.0926369Z * [new branch] gh/ZolotukhinM/468/orig -> origin/gh/ZolotukhinM/468/orig 2022-03-17T19:04:58.0927853Z * [new branch] gh/ZolotukhinM/469/base -> origin/gh/ZolotukhinM/469/base 2022-03-17T19:04:58.0929283Z * [new branch] gh/ZolotukhinM/469/head -> origin/gh/ZolotukhinM/469/head 2022-03-17T19:04:58.0930660Z * [new branch] gh/ZolotukhinM/469/orig -> origin/gh/ZolotukhinM/469/orig 2022-03-17T19:04:58.0932197Z * [new branch] gh/ZolotukhinM/470/base -> origin/gh/ZolotukhinM/470/base 2022-03-17T19:04:58.0933483Z * [new branch] gh/ZolotukhinM/470/head -> origin/gh/ZolotukhinM/470/head 2022-03-17T19:04:58.0934805Z * [new branch] gh/ZolotukhinM/470/orig -> origin/gh/ZolotukhinM/470/orig 2022-03-17T19:04:58.0936394Z * [new branch] gh/ZolotukhinM/471/base -> origin/gh/ZolotukhinM/471/base 2022-03-17T19:04:58.0937699Z * [new branch] gh/ZolotukhinM/471/head -> origin/gh/ZolotukhinM/471/head 2022-03-17T19:04:58.0939070Z * [new branch] gh/ZolotukhinM/471/orig -> origin/gh/ZolotukhinM/471/orig 2022-03-17T19:04:58.0940633Z * [new branch] gh/ZolotukhinM/472/base -> origin/gh/ZolotukhinM/472/base 2022-03-17T19:04:58.0942022Z * [new branch] gh/ZolotukhinM/472/head -> origin/gh/ZolotukhinM/472/head 2022-03-17T19:04:58.0943364Z * [new branch] gh/ZolotukhinM/472/orig -> origin/gh/ZolotukhinM/472/orig 2022-03-17T19:04:58.0945003Z * [new branch] gh/ZolotukhinM/473/base -> origin/gh/ZolotukhinM/473/base 2022-03-17T19:04:58.0946240Z * [new branch] gh/ZolotukhinM/473/head -> origin/gh/ZolotukhinM/473/head 2022-03-17T19:04:58.0947484Z * [new branch] gh/ZolotukhinM/473/orig -> origin/gh/ZolotukhinM/473/orig 2022-03-17T19:04:58.0949111Z * [new branch] gh/ZolotukhinM/474/base -> origin/gh/ZolotukhinM/474/base 2022-03-17T19:04:58.0950366Z * [new branch] gh/ZolotukhinM/474/head -> origin/gh/ZolotukhinM/474/head 2022-03-17T19:04:58.0951753Z * [new branch] gh/ZolotukhinM/474/orig -> origin/gh/ZolotukhinM/474/orig 2022-03-17T19:04:58.0953274Z * [new branch] gh/ZolotukhinM/475/base -> origin/gh/ZolotukhinM/475/base 2022-03-17T19:04:58.0954590Z * [new branch] gh/ZolotukhinM/475/head -> origin/gh/ZolotukhinM/475/head 2022-03-17T19:04:58.0955908Z * [new branch] gh/ZolotukhinM/475/orig -> origin/gh/ZolotukhinM/475/orig 2022-03-17T19:04:58.0957585Z * [new branch] gh/ZolotukhinM/482/base -> origin/gh/ZolotukhinM/482/base 2022-03-17T19:04:58.0958962Z * [new branch] gh/ZolotukhinM/482/head -> origin/gh/ZolotukhinM/482/head 2022-03-17T19:04:58.0960355Z * [new branch] gh/ZolotukhinM/482/orig -> origin/gh/ZolotukhinM/482/orig 2022-03-17T19:04:58.0962211Z * [new branch] gh/ZolotukhinM/483/base -> origin/gh/ZolotukhinM/483/base 2022-03-17T19:04:58.0963733Z * [new branch] gh/ZolotukhinM/483/head -> origin/gh/ZolotukhinM/483/head 2022-03-17T19:04:58.0965076Z * [new branch] gh/ZolotukhinM/483/orig -> origin/gh/ZolotukhinM/483/orig 2022-03-17T19:04:58.0966811Z * [new branch] gh/ZolotukhinM/484/base -> origin/gh/ZolotukhinM/484/base 2022-03-17T19:04:58.0968155Z * [new branch] gh/ZolotukhinM/484/head -> origin/gh/ZolotukhinM/484/head 2022-03-17T19:04:58.0969587Z * [new branch] gh/ZolotukhinM/484/orig -> origin/gh/ZolotukhinM/484/orig 2022-03-17T19:04:58.0971394Z * [new branch] gh/ZolotukhinM/485/base -> origin/gh/ZolotukhinM/485/base 2022-03-17T19:04:58.0972616Z * [new branch] gh/ZolotukhinM/485/head -> origin/gh/ZolotukhinM/485/head 2022-03-17T19:04:58.0973991Z * [new branch] gh/ZolotukhinM/485/orig -> origin/gh/ZolotukhinM/485/orig 2022-03-17T19:04:58.0975795Z * [new branch] gh/ZolotukhinM/489/base -> origin/gh/ZolotukhinM/489/base 2022-03-17T19:04:58.0977109Z * [new branch] gh/ZolotukhinM/489/head -> origin/gh/ZolotukhinM/489/head 2022-03-17T19:04:58.0978445Z * [new branch] gh/ZolotukhinM/489/orig -> origin/gh/ZolotukhinM/489/orig 2022-03-17T19:04:58.0980135Z * [new branch] gh/ZolotukhinM/503/base -> origin/gh/ZolotukhinM/503/base 2022-03-17T19:04:58.0981535Z * [new branch] gh/ZolotukhinM/503/head -> origin/gh/ZolotukhinM/503/head 2022-03-17T19:04:58.0983035Z * [new branch] gh/ZolotukhinM/503/orig -> origin/gh/ZolotukhinM/503/orig 2022-03-17T19:04:58.0984885Z * [new branch] gh/ZolotukhinM/506/base -> origin/gh/ZolotukhinM/506/base 2022-03-17T19:04:58.0985935Z * [new branch] gh/ZolotukhinM/506/head -> origin/gh/ZolotukhinM/506/head 2022-03-17T19:04:58.0987398Z * [new branch] gh/ZolotukhinM/506/orig -> origin/gh/ZolotukhinM/506/orig 2022-03-17T19:04:58.0989022Z * [new branch] gh/ZolotukhinM/66/base -> origin/gh/ZolotukhinM/66/base 2022-03-17T19:04:58.0990940Z * [new branch] gh/aazzolini/1/base -> origin/gh/aazzolini/1/base 2022-03-17T19:04:58.0992711Z * [new branch] gh/aazzolini/3/base -> origin/gh/aazzolini/3/base 2022-03-17T19:04:58.0994177Z * [new branch] gh/aazzolini/3/head -> origin/gh/aazzolini/3/head 2022-03-17T19:04:58.0995723Z * [new branch] gh/aazzolini/3/orig -> origin/gh/aazzolini/3/orig 2022-03-17T19:04:58.0997391Z * [new branch] gh/agolynski/1/base -> origin/gh/agolynski/1/base 2022-03-17T19:04:58.0998687Z * [new branch] gh/agolynski/1/head -> origin/gh/agolynski/1/head 2022-03-17T19:04:58.1000010Z * [new branch] gh/agolynski/1/orig -> origin/gh/agolynski/1/orig 2022-03-17T19:04:58.1001802Z * [new branch] gh/agolynski/10/base -> origin/gh/agolynski/10/base 2022-03-17T19:04:58.1004014Z * [new branch] gh/agolynski/10/head -> origin/gh/agolynski/10/head 2022-03-17T19:04:58.1005245Z * [new branch] gh/agolynski/10/orig -> origin/gh/agolynski/10/orig 2022-03-17T19:04:58.1006877Z * [new branch] gh/agolynski/11/base -> origin/gh/agolynski/11/base 2022-03-17T19:04:58.1008247Z * [new branch] gh/agolynski/11/head -> origin/gh/agolynski/11/head 2022-03-17T19:04:58.1009619Z * [new branch] gh/agolynski/11/orig -> origin/gh/agolynski/11/orig 2022-03-17T19:04:58.1011058Z * [new branch] gh/agolynski/2/base -> origin/gh/agolynski/2/base 2022-03-17T19:04:58.1012397Z * [new branch] gh/agolynski/2/head -> origin/gh/agolynski/2/head 2022-03-17T19:04:58.1013675Z * [new branch] gh/agolynski/2/orig -> origin/gh/agolynski/2/orig 2022-03-17T19:04:58.1015475Z * [new branch] gh/agolynski/21/base -> origin/gh/agolynski/21/base 2022-03-17T19:04:58.1016725Z * [new branch] gh/agolynski/21/head -> origin/gh/agolynski/21/head 2022-03-17T19:04:58.1018047Z * [new branch] gh/agolynski/21/orig -> origin/gh/agolynski/21/orig 2022-03-17T19:04:58.1019855Z * [new branch] gh/agolynski/22/base -> origin/gh/agolynski/22/base 2022-03-17T19:04:58.1021289Z * [new branch] gh/agolynski/22/head -> origin/gh/agolynski/22/head 2022-03-17T19:04:58.1022537Z * [new branch] gh/agolynski/22/orig -> origin/gh/agolynski/22/orig 2022-03-17T19:04:58.1024203Z * [new branch] gh/agolynski/6/base -> origin/gh/agolynski/6/base 2022-03-17T19:04:58.1025489Z * [new branch] gh/agolynski/6/head -> origin/gh/agolynski/6/head 2022-03-17T19:04:58.1026898Z * [new branch] gh/agolynski/6/orig -> origin/gh/agolynski/6/orig 2022-03-17T19:04:58.1028638Z * [new branch] gh/agolynski/8/base -> origin/gh/agolynski/8/base 2022-03-17T19:04:58.1029946Z * [new branch] gh/agolynski/8/orig -> origin/gh/agolynski/8/orig 2022-03-17T19:04:58.1031869Z * [new branch] gh/ailzhang/10/base -> origin/gh/ailzhang/10/base 2022-03-17T19:04:58.1033263Z * [new branch] gh/ailzhang/10/head -> origin/gh/ailzhang/10/head 2022-03-17T19:04:58.1034594Z * [new branch] gh/ailzhang/10/orig -> origin/gh/ailzhang/10/orig 2022-03-17T19:04:58.1036359Z * [new branch] gh/ailzhang/11/base -> origin/gh/ailzhang/11/base 2022-03-17T19:04:58.1037683Z * [new branch] gh/ailzhang/11/head -> origin/gh/ailzhang/11/head 2022-03-17T19:04:58.1039081Z * [new branch] gh/ailzhang/11/orig -> origin/gh/ailzhang/11/orig 2022-03-17T19:04:58.1040778Z * [new branch] gh/ailzhang/12/base -> origin/gh/ailzhang/12/base 2022-03-17T19:04:58.1042213Z * [new branch] gh/ailzhang/12/head -> origin/gh/ailzhang/12/head 2022-03-17T19:04:58.1043834Z * [new branch] gh/ailzhang/12/orig -> origin/gh/ailzhang/12/orig 2022-03-17T19:04:58.1045504Z * [new branch] gh/ailzhang/25/base -> origin/gh/ailzhang/25/base 2022-03-17T19:04:58.1046754Z * [new branch] gh/ailzhang/25/head -> origin/gh/ailzhang/25/head 2022-03-17T19:04:58.1048094Z * [new branch] gh/ailzhang/25/orig -> origin/gh/ailzhang/25/orig 2022-03-17T19:04:58.1049832Z * [new branch] gh/ailzhang/50/base -> origin/gh/ailzhang/50/base 2022-03-17T19:04:58.1051246Z * [new branch] gh/ailzhang/50/head -> origin/gh/ailzhang/50/head 2022-03-17T19:04:58.1052583Z * [new branch] gh/ailzhang/50/orig -> origin/gh/ailzhang/50/orig 2022-03-17T19:04:58.1054108Z * [new branch] gh/ailzhang/9/base -> origin/gh/ailzhang/9/base 2022-03-17T19:04:58.1055288Z * [new branch] gh/ailzhang/9/head -> origin/gh/ailzhang/9/head 2022-03-17T19:04:58.1056732Z * [new branch] gh/ailzhang/9/orig -> origin/gh/ailzhang/9/orig 2022-03-17T19:04:58.1058608Z * [new branch] gh/ajtulloch/1/base -> origin/gh/ajtulloch/1/base 2022-03-17T19:04:58.1059961Z * [new branch] gh/ajtulloch/1/head -> origin/gh/ajtulloch/1/head 2022-03-17T19:04:58.1061409Z * [new branch] gh/ajtulloch/2/base -> origin/gh/ajtulloch/2/base 2022-03-17T19:04:58.1062803Z * [new branch] gh/ajtulloch/2/head -> origin/gh/ajtulloch/2/head 2022-03-17T19:04:58.1064649Z * [new branch] gh/alanadakotashine/1/base -> origin/gh/alanadakotashine/1/base 2022-03-17T19:04:58.1066004Z * [new branch] gh/alanadakotashine/1/head -> origin/gh/alanadakotashine/1/head 2022-03-17T19:04:58.1067370Z * [new branch] gh/alanadakotashine/1/orig -> origin/gh/alanadakotashine/1/orig 2022-03-17T19:04:58.1069244Z * [new branch] gh/alanadakotashine/2/base -> origin/gh/alanadakotashine/2/base 2022-03-17T19:04:58.1070239Z * [new branch] gh/alanadakotashine/2/head -> origin/gh/alanadakotashine/2/head 2022-03-17T19:04:58.1071736Z * [new branch] gh/alanadakotashine/2/orig -> origin/gh/alanadakotashine/2/orig 2022-03-17T19:04:58.1073735Z * [new branch] gh/alanwaketan/10/base -> origin/gh/alanwaketan/10/base 2022-03-17T19:04:58.1075247Z * [new branch] gh/alanwaketan/10/head -> origin/gh/alanwaketan/10/head 2022-03-17T19:04:58.1076540Z * [new branch] gh/alanwaketan/10/orig -> origin/gh/alanwaketan/10/orig 2022-03-17T19:04:58.1078360Z * [new branch] gh/alanwaketan/15/base -> origin/gh/alanwaketan/15/base 2022-03-17T19:04:58.1079743Z * [new branch] gh/alanwaketan/15/orig -> origin/gh/alanwaketan/15/orig 2022-03-17T19:04:58.1081495Z * [new branch] gh/alanwaketan/4/base -> origin/gh/alanwaketan/4/base 2022-03-17T19:04:58.1083159Z * [new branch] gh/alanwaketan/4/head -> origin/gh/alanwaketan/4/head 2022-03-17T19:04:58.1084448Z * [new branch] gh/alanwaketan/4/orig -> origin/gh/alanwaketan/4/orig 2022-03-17T19:04:58.1086308Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2022-03-17T19:04:58.1087633Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2022-03-17T19:04:58.1089371Z * [new branch] gh/albanD/92/base -> origin/gh/albanD/92/base 2022-03-17T19:04:58.1090675Z * [new branch] gh/albanD/92/head -> origin/gh/albanD/92/head 2022-03-17T19:04:58.1091940Z * [new branch] gh/albanD/92/orig -> origin/gh/albanD/92/orig 2022-03-17T19:04:58.1093672Z * [new branch] gh/albanD/94/base -> origin/gh/albanD/94/base 2022-03-17T19:04:58.1095117Z * [new branch] gh/albanD/94/head -> origin/gh/albanD/94/head 2022-03-17T19:04:58.1096465Z * [new branch] gh/albanD/94/orig -> origin/gh/albanD/94/orig 2022-03-17T19:04:58.1098251Z * [new branch] gh/albanD/96/base -> origin/gh/albanD/96/base 2022-03-17T19:04:58.1099577Z * [new branch] gh/albanD/96/head -> origin/gh/albanD/96/head 2022-03-17T19:04:58.1100849Z * [new branch] gh/albanD/96/orig -> origin/gh/albanD/96/orig 2022-03-17T19:04:58.1102792Z * [new branch] gh/andrewor14/1/base -> origin/gh/andrewor14/1/base 2022-03-17T19:04:58.1104138Z * [new branch] gh/andrewor14/1/orig -> origin/gh/andrewor14/1/orig 2022-03-17T19:04:58.1105844Z * [new branch] gh/andrewor14/10/base -> origin/gh/andrewor14/10/base 2022-03-17T19:04:58.1107318Z * [new branch] gh/andrewor14/10/orig -> origin/gh/andrewor14/10/orig 2022-03-17T19:04:58.1109079Z * [new branch] gh/andrewor14/11/base -> origin/gh/andrewor14/11/base 2022-03-17T19:04:58.1110331Z * [new branch] gh/andrewor14/11/orig -> origin/gh/andrewor14/11/orig 2022-03-17T19:04:58.1112118Z * [new branch] gh/andrewor14/12/base -> origin/gh/andrewor14/12/base 2022-03-17T19:04:58.1113490Z * [new branch] gh/andrewor14/12/head -> origin/gh/andrewor14/12/head 2022-03-17T19:04:58.1114753Z * [new branch] gh/andrewor14/12/orig -> origin/gh/andrewor14/12/orig 2022-03-17T19:04:58.1116441Z * [new branch] gh/andrewor14/2/base -> origin/gh/andrewor14/2/base 2022-03-17T19:04:58.1117835Z * [new branch] gh/andrewor14/2/head -> origin/gh/andrewor14/2/head 2022-03-17T19:04:58.1119340Z * [new branch] gh/andrewor14/2/orig -> origin/gh/andrewor14/2/orig 2022-03-17T19:04:58.1121193Z * [new branch] gh/andrewor14/3/base -> origin/gh/andrewor14/3/base 2022-03-17T19:04:58.1123107Z * [new branch] gh/andrewor14/3/orig -> origin/gh/andrewor14/3/orig 2022-03-17T19:04:58.1124399Z * [new branch] gh/andrewor14/4/base -> origin/gh/andrewor14/4/base 2022-03-17T19:04:58.1125775Z * [new branch] gh/andrewor14/4/orig -> origin/gh/andrewor14/4/orig 2022-03-17T19:04:58.1127556Z * [new branch] gh/andrewor14/5/base -> origin/gh/andrewor14/5/base 2022-03-17T19:04:58.1128939Z * [new branch] gh/andrewor14/5/orig -> origin/gh/andrewor14/5/orig 2022-03-17T19:04:58.1130831Z * [new branch] gh/andrewor14/6/base -> origin/gh/andrewor14/6/base 2022-03-17T19:04:58.1132144Z * [new branch] gh/andrewor14/6/orig -> origin/gh/andrewor14/6/orig 2022-03-17T19:04:58.1133975Z * [new branch] gh/andrewor14/7/base -> origin/gh/andrewor14/7/base 2022-03-17T19:04:58.1135367Z * [new branch] gh/andrewor14/7/orig -> origin/gh/andrewor14/7/orig 2022-03-17T19:04:58.1137059Z * [new branch] gh/andrewor14/8/base -> origin/gh/andrewor14/8/base 2022-03-17T19:04:58.1138513Z * [new branch] gh/andrewor14/8/orig -> origin/gh/andrewor14/8/orig 2022-03-17T19:04:58.1140177Z * [new branch] gh/andrewor14/9/base -> origin/gh/andrewor14/9/base 2022-03-17T19:04:58.1141525Z * [new branch] gh/andrewor14/9/orig -> origin/gh/andrewor14/9/orig 2022-03-17T19:04:58.1143402Z * [new branch] gh/angelayi/18/base -> origin/gh/angelayi/18/base 2022-03-17T19:04:58.1144748Z * [new branch] gh/angelayi/18/head -> origin/gh/angelayi/18/head 2022-03-17T19:04:58.1146039Z * [new branch] gh/angelayi/18/orig -> origin/gh/angelayi/18/orig 2022-03-17T19:04:58.1148066Z * [new branch] gh/angelayi/33/base -> origin/gh/angelayi/33/base 2022-03-17T19:04:58.1149473Z * [new branch] gh/angelayi/33/head -> origin/gh/angelayi/33/head 2022-03-17T19:04:58.1150948Z * [new branch] gh/angelayi/33/orig -> origin/gh/angelayi/33/orig 2022-03-17T19:04:58.1152881Z * [new branch] gh/angelayi/34/base -> origin/gh/angelayi/34/base 2022-03-17T19:04:58.1154323Z * [new branch] gh/angelayi/34/head -> origin/gh/angelayi/34/head 2022-03-17T19:04:58.1155584Z * [new branch] gh/angelayi/34/orig -> origin/gh/angelayi/34/orig 2022-03-17T19:04:58.1157475Z * [new branch] gh/anijain2305/6/base -> origin/gh/anijain2305/6/base 2022-03-17T19:04:58.1158873Z * [new branch] gh/anijain2305/6/head -> origin/gh/anijain2305/6/head 2022-03-17T19:04:58.1160227Z * [new branch] gh/anijain2305/6/orig -> origin/gh/anijain2305/6/orig 2022-03-17T19:04:58.1162084Z * [new branch] gh/anjali411/100/base -> origin/gh/anjali411/100/base 2022-03-17T19:04:58.1164672Z * [new branch] gh/anjali411/100/head -> origin/gh/anjali411/100/head 2022-03-17T19:04:58.1166021Z * [new branch] gh/anjali411/100/orig -> origin/gh/anjali411/100/orig 2022-03-17T19:04:58.1167741Z * [new branch] gh/anjali411/106/base -> origin/gh/anjali411/106/base 2022-03-17T19:04:58.1169120Z * [new branch] gh/anjali411/106/head -> origin/gh/anjali411/106/head 2022-03-17T19:04:58.1170440Z * [new branch] gh/anjali411/106/orig -> origin/gh/anjali411/106/orig 2022-03-17T19:04:58.1172086Z * [new branch] gh/anjali411/124/base -> origin/gh/anjali411/124/base 2022-03-17T19:04:58.1173554Z * [new branch] gh/anjali411/124/head -> origin/gh/anjali411/124/head 2022-03-17T19:04:58.1174920Z * [new branch] gh/anjali411/124/orig -> origin/gh/anjali411/124/orig 2022-03-17T19:04:58.1176711Z * [new branch] gh/anjali411/128/base -> origin/gh/anjali411/128/base 2022-03-17T19:04:58.1178010Z * [new branch] gh/anjali411/128/head -> origin/gh/anjali411/128/head 2022-03-17T19:04:58.1179388Z * [new branch] gh/anjali411/128/orig -> origin/gh/anjali411/128/orig 2022-03-17T19:04:58.1181138Z * [new branch] gh/anjali411/13/base -> origin/gh/anjali411/13/base 2022-03-17T19:04:58.1182566Z * [new branch] gh/anjali411/13/head -> origin/gh/anjali411/13/head 2022-03-17T19:04:58.1183948Z * [new branch] gh/anjali411/13/orig -> origin/gh/anjali411/13/orig 2022-03-17T19:04:58.1185565Z * [new branch] gh/anjali411/130/base -> origin/gh/anjali411/130/base 2022-03-17T19:04:58.1186938Z * [new branch] gh/anjali411/130/head -> origin/gh/anjali411/130/head 2022-03-17T19:04:58.1188287Z * [new branch] gh/anjali411/130/orig -> origin/gh/anjali411/130/orig 2022-03-17T19:04:58.1190039Z * [new branch] gh/anjali411/14/base -> origin/gh/anjali411/14/base 2022-03-17T19:04:58.1191388Z * [new branch] gh/anjali411/14/orig -> origin/gh/anjali411/14/orig 2022-03-17T19:04:58.1193064Z * [new branch] gh/anjali411/145/base -> origin/gh/anjali411/145/base 2022-03-17T19:04:58.1194355Z * [new branch] gh/anjali411/145/head -> origin/gh/anjali411/145/head 2022-03-17T19:04:58.1195636Z * [new branch] gh/anjali411/145/orig -> origin/gh/anjali411/145/orig 2022-03-17T19:04:58.1197394Z * [new branch] gh/anjali411/153/base -> origin/gh/anjali411/153/base 2022-03-17T19:04:58.1198712Z * [new branch] gh/anjali411/153/head -> origin/gh/anjali411/153/head 2022-03-17T19:04:58.1200099Z * [new branch] gh/anjali411/153/orig -> origin/gh/anjali411/153/orig 2022-03-17T19:04:58.1201752Z * [new branch] gh/anjali411/155/base -> origin/gh/anjali411/155/base 2022-03-17T19:04:58.1203368Z * [new branch] gh/anjali411/155/head -> origin/gh/anjali411/155/head 2022-03-17T19:04:58.1204638Z * [new branch] gh/anjali411/155/orig -> origin/gh/anjali411/155/orig 2022-03-17T19:04:58.1206373Z * [new branch] gh/anjali411/156/base -> origin/gh/anjali411/156/base 2022-03-17T19:04:58.1207688Z * [new branch] gh/anjali411/156/head -> origin/gh/anjali411/156/head 2022-03-17T19:04:58.1208978Z * [new branch] gh/anjali411/156/orig -> origin/gh/anjali411/156/orig 2022-03-17T19:04:58.1210660Z * [new branch] gh/anjali411/159/base -> origin/gh/anjali411/159/base 2022-03-17T19:04:58.1211955Z * [new branch] gh/anjali411/159/head -> origin/gh/anjali411/159/head 2022-03-17T19:04:58.1213309Z * [new branch] gh/anjali411/159/orig -> origin/gh/anjali411/159/orig 2022-03-17T19:04:58.1215078Z * [new branch] gh/anjali411/160/base -> origin/gh/anjali411/160/base 2022-03-17T19:04:58.1216380Z * [new branch] gh/anjali411/160/head -> origin/gh/anjali411/160/head 2022-03-17T19:04:58.1217883Z * [new branch] gh/anjali411/160/orig -> origin/gh/anjali411/160/orig 2022-03-17T19:04:58.1219485Z * [new branch] gh/anjali411/19/base -> origin/gh/anjali411/19/base 2022-03-17T19:04:58.1220810Z * [new branch] gh/anjali411/19/head -> origin/gh/anjali411/19/head 2022-03-17T19:04:58.1222119Z * [new branch] gh/anjali411/19/orig -> origin/gh/anjali411/19/orig 2022-03-17T19:04:58.1223917Z * [new branch] gh/anjali411/22/base -> origin/gh/anjali411/22/base 2022-03-17T19:04:58.1225293Z * [new branch] gh/anjali411/22/head -> origin/gh/anjali411/22/head 2022-03-17T19:04:58.1226627Z * [new branch] gh/anjali411/22/orig -> origin/gh/anjali411/22/orig 2022-03-17T19:04:58.1228310Z * [new branch] gh/anjali411/25/base -> origin/gh/anjali411/25/base 2022-03-17T19:04:58.1229654Z * [new branch] gh/anjali411/25/head -> origin/gh/anjali411/25/head 2022-03-17T19:04:58.1230981Z * [new branch] gh/anjali411/25/orig -> origin/gh/anjali411/25/orig 2022-03-17T19:04:58.1232659Z * [new branch] gh/anjali411/26/base -> origin/gh/anjali411/26/base 2022-03-17T19:04:58.1234018Z * [new branch] gh/anjali411/26/head -> origin/gh/anjali411/26/head 2022-03-17T19:04:58.1235361Z * [new branch] gh/anjali411/26/orig -> origin/gh/anjali411/26/orig 2022-03-17T19:04:58.1237081Z * [new branch] gh/anjali411/31/base -> origin/gh/anjali411/31/base 2022-03-17T19:04:58.1238381Z * [new branch] gh/anjali411/31/head -> origin/gh/anjali411/31/head 2022-03-17T19:04:58.1239785Z * [new branch] gh/anjali411/31/orig -> origin/gh/anjali411/31/orig 2022-03-17T19:04:58.1241601Z * [new branch] gh/anjali411/68/base -> origin/gh/anjali411/68/base 2022-03-17T19:04:58.1243105Z * [new branch] gh/anjali411/68/head -> origin/gh/anjali411/68/head 2022-03-17T19:04:58.1244437Z * [new branch] gh/anjali411/68/orig -> origin/gh/anjali411/68/orig 2022-03-17T19:04:58.1246599Z * [new branch] gh/anjali411/77/base -> origin/gh/anjali411/77/base 2022-03-17T19:04:58.1247995Z * [new branch] gh/anjali411/77/head -> origin/gh/anjali411/77/head 2022-03-17T19:04:58.1249371Z * [new branch] gh/anjali411/77/orig -> origin/gh/anjali411/77/orig 2022-03-17T19:04:58.1251124Z * [new branch] gh/anjali411/80/base -> origin/gh/anjali411/80/base 2022-03-17T19:04:58.1252437Z * [new branch] gh/anjali411/80/orig -> origin/gh/anjali411/80/orig 2022-03-17T19:04:58.1254368Z * [new branch] gh/ansley/14/base -> origin/gh/ansley/14/base 2022-03-17T19:04:58.1255716Z * [new branch] gh/ansley/14/head -> origin/gh/ansley/14/head 2022-03-17T19:04:58.1257060Z * [new branch] gh/ansley/14/orig -> origin/gh/ansley/14/orig 2022-03-17T19:04:58.1258854Z * [new branch] gh/ansley/24/base -> origin/gh/ansley/24/base 2022-03-17T19:04:58.1260190Z * [new branch] gh/ansley/24/head -> origin/gh/ansley/24/head 2022-03-17T19:04:58.1261511Z * [new branch] gh/ansley/24/orig -> origin/gh/ansley/24/orig 2022-03-17T19:04:58.1263323Z * [new branch] gh/ansley/25/base -> origin/gh/ansley/25/base 2022-03-17T19:04:58.1264714Z * [new branch] gh/ansley/25/head -> origin/gh/ansley/25/head 2022-03-17T19:04:58.1265993Z * [new branch] gh/ansley/25/orig -> origin/gh/ansley/25/orig 2022-03-17T19:04:58.1267715Z * [new branch] gh/ansley/26/base -> origin/gh/ansley/26/base 2022-03-17T19:04:58.1269008Z * [new branch] gh/ansley/26/head -> origin/gh/ansley/26/head 2022-03-17T19:04:58.1270345Z * [new branch] gh/ansley/26/orig -> origin/gh/ansley/26/orig 2022-03-17T19:04:58.1272079Z * [new branch] gh/ansley/27/base -> origin/gh/ansley/27/base 2022-03-17T19:04:58.1273349Z * [new branch] gh/ansley/27/head -> origin/gh/ansley/27/head 2022-03-17T19:04:58.1274681Z * [new branch] gh/ansley/27/orig -> origin/gh/ansley/27/orig 2022-03-17T19:04:58.1276444Z * [new branch] gh/ansley/28/base -> origin/gh/ansley/28/base 2022-03-17T19:04:58.1277868Z * [new branch] gh/ansley/28/head -> origin/gh/ansley/28/head 2022-03-17T19:04:58.1279129Z * [new branch] gh/ansley/28/orig -> origin/gh/ansley/28/orig 2022-03-17T19:04:58.1280829Z * [new branch] gh/ansley/30/base -> origin/gh/ansley/30/base 2022-03-17T19:04:58.1282256Z * [new branch] gh/ansley/30/head -> origin/gh/ansley/30/head 2022-03-17T19:04:58.1284700Z * [new branch] gh/ansley/30/orig -> origin/gh/ansley/30/orig 2022-03-17T19:04:58.1286285Z * [new branch] gh/ansley/35/base -> origin/gh/ansley/35/base 2022-03-17T19:04:58.1287640Z * [new branch] gh/ansley/35/head -> origin/gh/ansley/35/head 2022-03-17T19:04:58.1288989Z * [new branch] gh/ansley/35/orig -> origin/gh/ansley/35/orig 2022-03-17T19:04:58.1290745Z * [new branch] gh/ansley/36/base -> origin/gh/ansley/36/base 2022-03-17T19:04:58.1292055Z * [new branch] gh/ansley/36/head -> origin/gh/ansley/36/head 2022-03-17T19:04:58.1293441Z * [new branch] gh/ansley/36/orig -> origin/gh/ansley/36/orig 2022-03-17T19:04:58.1295282Z * [new branch] gh/ansley/38/base -> origin/gh/ansley/38/base 2022-03-17T19:04:58.1296781Z * [new branch] gh/ansley/38/head -> origin/gh/ansley/38/head 2022-03-17T19:04:58.1298079Z * [new branch] gh/ansley/38/orig -> origin/gh/ansley/38/orig 2022-03-17T19:04:58.1299829Z * [new branch] gh/ansley/41/base -> origin/gh/ansley/41/base 2022-03-17T19:04:58.1301034Z * [new branch] gh/ansley/41/head -> origin/gh/ansley/41/head 2022-03-17T19:04:58.1302370Z * [new branch] gh/ansley/41/orig -> origin/gh/ansley/41/orig 2022-03-17T19:04:58.1303968Z * [new branch] gh/ansley/43/base -> origin/gh/ansley/43/base 2022-03-17T19:04:58.1305322Z * [new branch] gh/ansley/43/head -> origin/gh/ansley/43/head 2022-03-17T19:04:58.1306655Z * [new branch] gh/ansley/43/orig -> origin/gh/ansley/43/orig 2022-03-17T19:04:58.1308379Z * [new branch] gh/ansley/45/base -> origin/gh/ansley/45/base 2022-03-17T19:04:58.1309657Z * [new branch] gh/ansley/45/head -> origin/gh/ansley/45/head 2022-03-17T19:04:58.1310962Z * [new branch] gh/ansley/45/orig -> origin/gh/ansley/45/orig 2022-03-17T19:04:58.1312717Z * [new branch] gh/ansley/47/base -> origin/gh/ansley/47/base 2022-03-17T19:04:58.1314078Z * [new branch] gh/ansley/47/head -> origin/gh/ansley/47/head 2022-03-17T19:04:58.1315382Z * [new branch] gh/ansley/47/orig -> origin/gh/ansley/47/orig 2022-03-17T19:04:58.1316996Z * [new branch] gh/ansley/48/base -> origin/gh/ansley/48/base 2022-03-17T19:04:58.1318310Z * [new branch] gh/ansley/48/head -> origin/gh/ansley/48/head 2022-03-17T19:04:58.1319809Z * [new branch] gh/ansley/48/orig -> origin/gh/ansley/48/orig 2022-03-17T19:04:58.1321488Z * [new branch] gh/ansley/51/base -> origin/gh/ansley/51/base 2022-03-17T19:04:58.1323486Z * [new branch] gh/ansley/51/head -> origin/gh/ansley/51/head 2022-03-17T19:04:58.1324928Z * [new branch] gh/ansley/51/orig -> origin/gh/ansley/51/orig 2022-03-17T19:04:58.1326949Z * [new branch] gh/ansley/54/base -> origin/gh/ansley/54/base 2022-03-17T19:04:58.1328420Z * [new branch] gh/ansley/54/head -> origin/gh/ansley/54/head 2022-03-17T19:04:58.1329705Z * [new branch] gh/ansley/54/orig -> origin/gh/ansley/54/orig 2022-03-17T19:04:58.1331669Z * [new branch] gh/ansley/55/base -> origin/gh/ansley/55/base 2022-03-17T19:04:58.1333405Z * [new branch] gh/ansley/55/head -> origin/gh/ansley/55/head 2022-03-17T19:04:58.1334906Z * [new branch] gh/ansley/55/orig -> origin/gh/ansley/55/orig 2022-03-17T19:04:58.1336624Z * [new branch] gh/ansley/56/base -> origin/gh/ansley/56/base 2022-03-17T19:04:58.1338059Z * [new branch] gh/ansley/56/head -> origin/gh/ansley/56/head 2022-03-17T19:04:58.1339396Z * [new branch] gh/ansley/56/orig -> origin/gh/ansley/56/orig 2022-03-17T19:04:58.1341048Z * [new branch] gh/ansley/57/base -> origin/gh/ansley/57/base 2022-03-17T19:04:58.1342454Z * [new branch] gh/ansley/57/head -> origin/gh/ansley/57/head 2022-03-17T19:04:58.1343777Z * [new branch] gh/ansley/57/orig -> origin/gh/ansley/57/orig 2022-03-17T19:04:58.1345305Z * [new branch] gh/ansley/58/base -> origin/gh/ansley/58/base 2022-03-17T19:04:58.1346551Z * [new branch] gh/ansley/58/head -> origin/gh/ansley/58/head 2022-03-17T19:04:58.1347862Z * [new branch] gh/ansley/58/orig -> origin/gh/ansley/58/orig 2022-03-17T19:04:58.1349366Z * [new branch] gh/ansley/59/base -> origin/gh/ansley/59/base 2022-03-17T19:04:58.1350779Z * [new branch] gh/ansley/59/head -> origin/gh/ansley/59/head 2022-03-17T19:04:58.1352059Z * [new branch] gh/ansley/59/orig -> origin/gh/ansley/59/orig 2022-03-17T19:04:58.1354014Z * [new branch] gh/ansleyadelaide/1/base -> origin/gh/ansleyadelaide/1/base 2022-03-17T19:04:58.1355359Z * [new branch] gh/ansleyadelaide/1/orig -> origin/gh/ansleyadelaide/1/orig 2022-03-17T19:04:58.1357104Z * [new branch] gh/ansleyadelaide/10/base -> origin/gh/ansleyadelaide/10/base 2022-03-17T19:04:58.1358425Z * [new branch] gh/ansleyadelaide/10/orig -> origin/gh/ansleyadelaide/10/orig 2022-03-17T19:04:58.1360091Z * [new branch] gh/ansleyadelaide/11/base -> origin/gh/ansleyadelaide/11/base 2022-03-17T19:04:58.1361400Z * [new branch] gh/ansleyadelaide/11/head -> origin/gh/ansleyadelaide/11/head 2022-03-17T19:04:58.1363687Z * [new branch] gh/ansleyadelaide/14/base -> origin/gh/ansleyadelaide/14/base 2022-03-17T19:04:58.1364898Z * [new branch] gh/ansleyadelaide/14/orig -> origin/gh/ansleyadelaide/14/orig 2022-03-17T19:04:58.1366724Z * [new branch] gh/ansleyadelaide/18/base -> origin/gh/ansleyadelaide/18/base 2022-03-17T19:04:58.1368104Z * [new branch] gh/ansleyadelaide/18/head -> origin/gh/ansleyadelaide/18/head 2022-03-17T19:04:58.1369477Z * [new branch] gh/ansleyadelaide/18/orig -> origin/gh/ansleyadelaide/18/orig 2022-03-17T19:04:58.1370917Z * [new branch] gh/ansleyadelaide/2/base -> origin/gh/ansleyadelaide/2/base 2022-03-17T19:04:58.1372274Z * [new branch] gh/ansleyadelaide/2/head -> origin/gh/ansleyadelaide/2/head 2022-03-17T19:04:58.1373528Z * [new branch] gh/ansleyadelaide/2/orig -> origin/gh/ansleyadelaide/2/orig 2022-03-17T19:04:58.1375261Z * [new branch] gh/ansleyadelaide/22/base -> origin/gh/ansleyadelaide/22/base 2022-03-17T19:04:58.1376571Z * [new branch] gh/ansleyadelaide/22/orig -> origin/gh/ansleyadelaide/22/orig 2022-03-17T19:04:58.1378378Z * [new branch] gh/ansleyadelaide/26/base -> origin/gh/ansleyadelaide/26/base 2022-03-17T19:04:58.1379768Z * [new branch] gh/ansleyadelaide/26/orig -> origin/gh/ansleyadelaide/26/orig 2022-03-17T19:04:58.1381732Z * [new branch] gh/ansleyadelaide/28/base -> origin/gh/ansleyadelaide/28/base 2022-03-17T19:04:58.1383102Z * [new branch] gh/ansleyadelaide/28/orig -> origin/gh/ansleyadelaide/28/orig 2022-03-17T19:04:58.1384718Z * [new branch] gh/ansleyadelaide/29/base -> origin/gh/ansleyadelaide/29/base 2022-03-17T19:04:58.1386062Z * [new branch] gh/ansleyadelaide/29/orig -> origin/gh/ansleyadelaide/29/orig 2022-03-17T19:04:58.1387758Z * [new branch] gh/ansleyadelaide/3/base -> origin/gh/ansleyadelaide/3/base 2022-03-17T19:04:58.1389057Z * [new branch] gh/ansleyadelaide/3/head -> origin/gh/ansleyadelaide/3/head 2022-03-17T19:04:58.1390390Z * [new branch] gh/ansleyadelaide/3/orig -> origin/gh/ansleyadelaide/3/orig 2022-03-17T19:04:58.1392071Z * [new branch] gh/ansleyadelaide/31/base -> origin/gh/ansleyadelaide/31/base 2022-03-17T19:04:58.1393499Z * [new branch] gh/ansleyadelaide/31/orig -> origin/gh/ansleyadelaide/31/orig 2022-03-17T19:04:58.1395180Z * [new branch] gh/ansleyadelaide/37/base -> origin/gh/ansleyadelaide/37/base 2022-03-17T19:04:58.1396466Z * [new branch] gh/ansleyadelaide/37/head -> origin/gh/ansleyadelaide/37/head 2022-03-17T19:04:58.1397787Z * [new branch] gh/ansleyadelaide/37/orig -> origin/gh/ansleyadelaide/37/orig 2022-03-17T19:04:58.1399681Z * [new branch] gh/ansleyadelaide/4/base -> origin/gh/ansleyadelaide/4/base 2022-03-17T19:04:58.1400971Z * [new branch] gh/ansleyadelaide/4/head -> origin/gh/ansleyadelaide/4/head 2022-03-17T19:04:58.1402374Z * [new branch] gh/ansleyadelaide/4/orig -> origin/gh/ansleyadelaide/4/orig 2022-03-17T19:04:58.1404950Z * [new branch] gh/ansleyadelaide/43/base -> origin/gh/ansleyadelaide/43/base 2022-03-17T19:04:58.1406631Z * [new branch] gh/ansleyadelaide/9/base -> origin/gh/ansleyadelaide/9/base 2022-03-17T19:04:58.1407911Z * [new branch] gh/ansleyadelaide/9/orig -> origin/gh/ansleyadelaide/9/orig 2022-03-17T19:04:58.1409861Z * [new branch] gh/aocsa/1/base -> origin/gh/aocsa/1/base 2022-03-17T19:04:58.1411265Z * [new branch] gh/aocsa/1/orig -> origin/gh/aocsa/1/orig 2022-03-17T19:04:58.1413082Z * [new branch] gh/aocsa/20/base -> origin/gh/aocsa/20/base 2022-03-17T19:04:58.1414495Z * [new branch] gh/aocsa/20/head -> origin/gh/aocsa/20/head 2022-03-17T19:04:58.1416349Z * [new branch] gh/asalioufb/1/base -> origin/gh/asalioufb/1/base 2022-03-17T19:04:58.1417703Z * [new branch] gh/asalioufb/1/head -> origin/gh/asalioufb/1/head 2022-03-17T19:04:58.1419544Z * [new branch] gh/awgu/14/base -> origin/gh/awgu/14/base 2022-03-17T19:04:58.1420852Z * [new branch] gh/awgu/14/head -> origin/gh/awgu/14/head 2022-03-17T19:04:58.1422240Z * [new branch] gh/awgu/14/orig -> origin/gh/awgu/14/orig 2022-03-17T19:04:58.1423704Z * [new branch] gh/awgu/15/base -> origin/gh/awgu/15/base 2022-03-17T19:04:58.1425116Z * [new branch] gh/awgu/15/head -> origin/gh/awgu/15/head 2022-03-17T19:04:58.1426413Z * [new branch] gh/awgu/15/orig -> origin/gh/awgu/15/orig 2022-03-17T19:04:58.1427937Z * [new branch] gh/awgu/16/base -> origin/gh/awgu/16/base 2022-03-17T19:04:58.1429277Z * [new branch] gh/awgu/16/head -> origin/gh/awgu/16/head 2022-03-17T19:04:58.1430637Z * [new branch] gh/awgu/16/orig -> origin/gh/awgu/16/orig 2022-03-17T19:04:58.1432540Z * [new branch] gh/axitkhurana/10/base -> origin/gh/axitkhurana/10/base 2022-03-17T19:04:58.1433906Z * [new branch] gh/axitkhurana/10/head -> origin/gh/axitkhurana/10/head 2022-03-17T19:04:58.1435315Z * [new branch] gh/axitkhurana/10/orig -> origin/gh/axitkhurana/10/orig 2022-03-17T19:04:58.1437050Z * [new branch] gh/axitkhurana/19/base -> origin/gh/axitkhurana/19/base 2022-03-17T19:04:58.1438512Z * [new branch] gh/axitkhurana/19/head -> origin/gh/axitkhurana/19/head 2022-03-17T19:04:58.1439886Z * [new branch] gh/axitkhurana/19/orig -> origin/gh/axitkhurana/19/orig 2022-03-17T19:04:58.1441578Z * [new branch] gh/axitkhurana/21/base -> origin/gh/axitkhurana/21/base 2022-03-17T19:04:58.1443246Z * [new branch] gh/axitkhurana/21/head -> origin/gh/axitkhurana/21/head 2022-03-17T19:04:58.1444475Z * [new branch] gh/axitkhurana/21/orig -> origin/gh/axitkhurana/21/orig 2022-03-17T19:04:58.1446306Z * [new branch] gh/axitkhurana/23/base -> origin/gh/axitkhurana/23/base 2022-03-17T19:04:58.1447661Z * [new branch] gh/axitkhurana/23/orig -> origin/gh/axitkhurana/23/orig 2022-03-17T19:04:58.1449419Z * [new branch] gh/axitkhurana/6/base -> origin/gh/axitkhurana/6/base 2022-03-17T19:04:58.1450816Z * [new branch] gh/axitkhurana/6/head -> origin/gh/axitkhurana/6/head 2022-03-17T19:04:58.1452130Z * [new branch] gh/axitkhurana/6/orig -> origin/gh/axitkhurana/6/orig 2022-03-17T19:04:58.1454024Z * [new branch] gh/b-koopman/11/base -> origin/gh/b-koopman/11/base 2022-03-17T19:04:58.1455438Z * [new branch] gh/b-koopman/11/head -> origin/gh/b-koopman/11/head 2022-03-17T19:04:58.1456880Z * [new branch] gh/b-koopman/11/orig -> origin/gh/b-koopman/11/orig 2022-03-17T19:04:58.1458614Z * [new branch] gh/b-koopman/13/base -> origin/gh/b-koopman/13/base 2022-03-17T19:04:58.1460094Z * [new branch] gh/b-koopman/13/head -> origin/gh/b-koopman/13/head 2022-03-17T19:04:58.1461463Z * [new branch] gh/b-koopman/13/orig -> origin/gh/b-koopman/13/orig 2022-03-17T19:04:58.1463010Z * [new branch] gh/b-koopman/20/base -> origin/gh/b-koopman/20/base 2022-03-17T19:04:58.1464288Z * [new branch] gh/b-koopman/20/head -> origin/gh/b-koopman/20/head 2022-03-17T19:04:58.1465578Z * [new branch] gh/b-koopman/20/orig -> origin/gh/b-koopman/20/orig 2022-03-17T19:04:58.1467574Z * [new branch] gh/banitag1/1/base -> origin/gh/banitag1/1/base 2022-03-17T19:04:58.1469106Z * [new branch] gh/banitag1/1/head -> origin/gh/banitag1/1/head 2022-03-17T19:04:58.1470795Z * [new branch] gh/banitag1/1/orig -> origin/gh/banitag1/1/orig 2022-03-17T19:04:58.1472826Z * [new branch] gh/bdhirsh/108/base -> origin/gh/bdhirsh/108/base 2022-03-17T19:04:58.1474384Z * [new branch] gh/bdhirsh/108/head -> origin/gh/bdhirsh/108/head 2022-03-17T19:04:58.1475756Z * [new branch] gh/bdhirsh/108/orig -> origin/gh/bdhirsh/108/orig 2022-03-17T19:04:58.1478117Z * [new branch] gh/bdhirsh/109/base -> origin/gh/bdhirsh/109/base 2022-03-17T19:04:58.1479791Z * [new branch] gh/bdhirsh/109/head -> origin/gh/bdhirsh/109/head 2022-03-17T19:04:58.1481112Z * [new branch] gh/bdhirsh/109/orig -> origin/gh/bdhirsh/109/orig 2022-03-17T19:04:58.1482963Z * [new branch] gh/bdhirsh/111/base -> origin/gh/bdhirsh/111/base 2022-03-17T19:04:58.1484235Z * [new branch] gh/bdhirsh/111/head -> origin/gh/bdhirsh/111/head 2022-03-17T19:04:58.1485535Z * [new branch] gh/bdhirsh/111/orig -> origin/gh/bdhirsh/111/orig 2022-03-17T19:04:58.1487192Z * [new branch] gh/bdhirsh/114/base -> origin/gh/bdhirsh/114/base 2022-03-17T19:04:58.1488490Z * [new branch] gh/bdhirsh/114/head -> origin/gh/bdhirsh/114/head 2022-03-17T19:04:58.1489798Z * [new branch] gh/bdhirsh/114/orig -> origin/gh/bdhirsh/114/orig 2022-03-17T19:04:58.1491728Z * [new branch] gh/bdhirsh/115/base -> origin/gh/bdhirsh/115/base 2022-03-17T19:04:58.1493148Z * [new branch] gh/bdhirsh/115/head -> origin/gh/bdhirsh/115/head 2022-03-17T19:04:58.1494458Z * [new branch] gh/bdhirsh/115/orig -> origin/gh/bdhirsh/115/orig 2022-03-17T19:04:58.1496194Z * [new branch] gh/bdhirsh/118/base -> origin/gh/bdhirsh/118/base 2022-03-17T19:04:58.1497528Z * [new branch] gh/bdhirsh/118/head -> origin/gh/bdhirsh/118/head 2022-03-17T19:04:58.1498877Z * [new branch] gh/bdhirsh/118/orig -> origin/gh/bdhirsh/118/orig 2022-03-17T19:04:58.1500562Z * [new branch] gh/bdhirsh/119/base -> origin/gh/bdhirsh/119/base 2022-03-17T19:04:58.1501903Z * [new branch] gh/bdhirsh/119/head -> origin/gh/bdhirsh/119/head 2022-03-17T19:04:58.1503215Z * [new branch] gh/bdhirsh/119/orig -> origin/gh/bdhirsh/119/orig 2022-03-17T19:04:58.1504901Z * [new branch] gh/bdhirsh/128/base -> origin/gh/bdhirsh/128/base 2022-03-17T19:04:58.1506234Z * [new branch] gh/bdhirsh/128/head -> origin/gh/bdhirsh/128/head 2022-03-17T19:04:58.1507544Z * [new branch] gh/bdhirsh/128/orig -> origin/gh/bdhirsh/128/orig 2022-03-17T19:04:58.1509256Z * [new branch] gh/bdhirsh/131/base -> origin/gh/bdhirsh/131/base 2022-03-17T19:04:58.1510700Z * [new branch] gh/bdhirsh/131/head -> origin/gh/bdhirsh/131/head 2022-03-17T19:04:58.1512042Z * [new branch] gh/bdhirsh/131/orig -> origin/gh/bdhirsh/131/orig 2022-03-17T19:04:58.1513938Z * [new branch] gh/bdhirsh/132/base -> origin/gh/bdhirsh/132/base 2022-03-17T19:04:58.1515491Z * [new branch] gh/bdhirsh/132/head -> origin/gh/bdhirsh/132/head 2022-03-17T19:04:58.1516866Z * [new branch] gh/bdhirsh/132/orig -> origin/gh/bdhirsh/132/orig 2022-03-17T19:04:58.1518888Z * [new branch] gh/bdhirsh/133/base -> origin/gh/bdhirsh/133/base 2022-03-17T19:04:58.1520292Z * [new branch] gh/bdhirsh/133/head -> origin/gh/bdhirsh/133/head 2022-03-17T19:04:58.1521668Z * [new branch] gh/bdhirsh/133/orig -> origin/gh/bdhirsh/133/orig 2022-03-17T19:04:58.1523871Z * [new branch] gh/bdhirsh/134/base -> origin/gh/bdhirsh/134/base 2022-03-17T19:04:58.1525379Z * [new branch] gh/bdhirsh/134/head -> origin/gh/bdhirsh/134/head 2022-03-17T19:04:58.1526736Z * [new branch] gh/bdhirsh/134/orig -> origin/gh/bdhirsh/134/orig 2022-03-17T19:04:58.1528642Z * [new branch] gh/bdhirsh/136/base -> origin/gh/bdhirsh/136/base 2022-03-17T19:04:58.1530164Z * [new branch] gh/bdhirsh/136/head -> origin/gh/bdhirsh/136/head 2022-03-17T19:04:58.1531666Z * [new branch] gh/bdhirsh/136/orig -> origin/gh/bdhirsh/136/orig 2022-03-17T19:04:58.1533510Z * [new branch] gh/bdhirsh/138/base -> origin/gh/bdhirsh/138/base 2022-03-17T19:04:58.1535112Z * [new branch] gh/bdhirsh/138/head -> origin/gh/bdhirsh/138/head 2022-03-17T19:04:58.1536596Z * [new branch] gh/bdhirsh/138/orig -> origin/gh/bdhirsh/138/orig 2022-03-17T19:04:58.1538382Z * [new branch] gh/bdhirsh/139/base -> origin/gh/bdhirsh/139/base 2022-03-17T19:04:58.1539759Z * [new branch] gh/bdhirsh/139/head -> origin/gh/bdhirsh/139/head 2022-03-17T19:04:58.1541230Z * [new branch] gh/bdhirsh/139/orig -> origin/gh/bdhirsh/139/orig 2022-03-17T19:04:58.1543119Z * [new branch] gh/bdhirsh/140/base -> origin/gh/bdhirsh/140/base 2022-03-17T19:04:58.1545945Z * [new branch] gh/bdhirsh/140/head -> origin/gh/bdhirsh/140/head 2022-03-17T19:04:58.1547335Z * [new branch] gh/bdhirsh/140/orig -> origin/gh/bdhirsh/140/orig 2022-03-17T19:04:58.1549308Z * [new branch] gh/bdhirsh/142/base -> origin/gh/bdhirsh/142/base 2022-03-17T19:04:58.1551061Z * [new branch] gh/bdhirsh/142/head -> origin/gh/bdhirsh/142/head 2022-03-17T19:04:58.1552627Z * [new branch] gh/bdhirsh/142/orig -> origin/gh/bdhirsh/142/orig 2022-03-17T19:04:58.1554223Z * [new branch] gh/bdhirsh/144/base -> origin/gh/bdhirsh/144/base 2022-03-17T19:04:58.1555724Z * [new branch] gh/bdhirsh/144/head -> origin/gh/bdhirsh/144/head 2022-03-17T19:04:58.1557376Z * [new branch] gh/bdhirsh/144/orig -> origin/gh/bdhirsh/144/orig 2022-03-17T19:04:58.1559788Z * [new branch] gh/bdhirsh/146/base -> origin/gh/bdhirsh/146/base 2022-03-17T19:04:58.1561365Z * [new branch] gh/bdhirsh/146/head -> origin/gh/bdhirsh/146/head 2022-03-17T19:04:58.1563353Z * [new branch] gh/bdhirsh/146/orig -> origin/gh/bdhirsh/146/orig 2022-03-17T19:04:58.1570986Z * [new branch] gh/bdhirsh/154/base -> origin/gh/bdhirsh/154/base 2022-03-17T19:04:58.1574004Z * [new branch] gh/bdhirsh/154/head -> origin/gh/bdhirsh/154/head 2022-03-17T19:04:58.1576970Z * [new branch] gh/bdhirsh/154/orig -> origin/gh/bdhirsh/154/orig 2022-03-17T19:04:58.1580704Z * [new branch] gh/bdhirsh/162/base -> origin/gh/bdhirsh/162/base 2022-03-17T19:04:58.1583859Z * [new branch] gh/bdhirsh/162/head -> origin/gh/bdhirsh/162/head 2022-03-17T19:04:58.1586899Z * [new branch] gh/bdhirsh/162/orig -> origin/gh/bdhirsh/162/orig 2022-03-17T19:04:58.1590592Z * [new branch] gh/bdhirsh/163/base -> origin/gh/bdhirsh/163/base 2022-03-17T19:04:58.1594466Z * [new branch] gh/bdhirsh/163/head -> origin/gh/bdhirsh/163/head 2022-03-17T19:04:58.1597387Z * [new branch] gh/bdhirsh/163/orig -> origin/gh/bdhirsh/163/orig 2022-03-17T19:04:58.1601546Z * [new branch] gh/bdhirsh/167/base -> origin/gh/bdhirsh/167/base 2022-03-17T19:04:58.1605422Z * [new branch] gh/bdhirsh/167/head -> origin/gh/bdhirsh/167/head 2022-03-17T19:04:58.1608319Z * [new branch] gh/bdhirsh/167/orig -> origin/gh/bdhirsh/167/orig 2022-03-17T19:04:58.1612024Z * [new branch] gh/bdhirsh/172/base -> origin/gh/bdhirsh/172/base 2022-03-17T19:04:58.1614973Z * [new branch] gh/bdhirsh/172/head -> origin/gh/bdhirsh/172/head 2022-03-17T19:04:58.1617848Z * [new branch] gh/bdhirsh/172/orig -> origin/gh/bdhirsh/172/orig 2022-03-17T19:04:58.1621631Z * [new branch] gh/bdhirsh/174/base -> origin/gh/bdhirsh/174/base 2022-03-17T19:04:58.1624600Z * [new branch] gh/bdhirsh/174/head -> origin/gh/bdhirsh/174/head 2022-03-17T19:04:58.1628247Z * [new branch] gh/bdhirsh/177/base -> origin/gh/bdhirsh/177/base 2022-03-17T19:04:58.1631084Z * [new branch] gh/bdhirsh/177/head -> origin/gh/bdhirsh/177/head 2022-03-17T19:04:58.1634044Z * [new branch] gh/bdhirsh/177/orig -> origin/gh/bdhirsh/177/orig 2022-03-17T19:04:58.1637973Z * [new branch] gh/bdhirsh/178/base -> origin/gh/bdhirsh/178/base 2022-03-17T19:04:58.1640720Z * [new branch] gh/bdhirsh/178/head -> origin/gh/bdhirsh/178/head 2022-03-17T19:04:58.1643791Z * [new branch] gh/bdhirsh/178/orig -> origin/gh/bdhirsh/178/orig 2022-03-17T19:04:58.1647571Z * [new branch] gh/bdhirsh/182/base -> origin/gh/bdhirsh/182/base 2022-03-17T19:04:58.1650433Z * [new branch] gh/bdhirsh/182/head -> origin/gh/bdhirsh/182/head 2022-03-17T19:04:58.1653354Z * [new branch] gh/bdhirsh/182/orig -> origin/gh/bdhirsh/182/orig 2022-03-17T19:04:58.1657246Z * [new branch] gh/bdhirsh/183/base -> origin/gh/bdhirsh/183/base 2022-03-17T19:04:58.1660307Z * [new branch] gh/bdhirsh/183/head -> origin/gh/bdhirsh/183/head 2022-03-17T19:04:58.1663168Z * [new branch] gh/bdhirsh/183/orig -> origin/gh/bdhirsh/183/orig 2022-03-17T19:04:58.1666866Z * [new branch] gh/bdhirsh/184/base -> origin/gh/bdhirsh/184/base 2022-03-17T19:04:58.1669836Z * [new branch] gh/bdhirsh/184/head -> origin/gh/bdhirsh/184/head 2022-03-17T19:04:58.1672919Z * [new branch] gh/bdhirsh/184/orig -> origin/gh/bdhirsh/184/orig 2022-03-17T19:04:58.1676658Z * [new branch] gh/bdhirsh/185/base -> origin/gh/bdhirsh/185/base 2022-03-17T19:04:58.1679760Z * [new branch] gh/bdhirsh/185/head -> origin/gh/bdhirsh/185/head 2022-03-17T19:04:58.1682630Z * [new branch] gh/bdhirsh/185/orig -> origin/gh/bdhirsh/185/orig 2022-03-17T19:04:58.1686170Z * [new branch] gh/bdhirsh/186/base -> origin/gh/bdhirsh/186/base 2022-03-17T19:04:58.1688947Z * [new branch] gh/bdhirsh/186/head -> origin/gh/bdhirsh/186/head 2022-03-17T19:04:58.1691859Z * [new branch] gh/bdhirsh/186/orig -> origin/gh/bdhirsh/186/orig 2022-03-17T19:04:58.1695016Z * [new branch] gh/bdhirsh/187/base -> origin/gh/bdhirsh/187/base 2022-03-17T19:04:58.1697859Z * [new branch] gh/bdhirsh/187/head -> origin/gh/bdhirsh/187/head 2022-03-17T19:04:58.1700728Z * [new branch] gh/bdhirsh/187/orig -> origin/gh/bdhirsh/187/orig 2022-03-17T19:04:58.1704438Z * [new branch] gh/bdhirsh/25/base -> origin/gh/bdhirsh/25/base 2022-03-17T19:04:58.1708754Z * [new branch] gh/bdhirsh/25/head -> origin/gh/bdhirsh/25/head 2022-03-17T19:04:58.1711744Z * [new branch] gh/bdhirsh/25/orig -> origin/gh/bdhirsh/25/orig 2022-03-17T19:04:58.1715363Z * [new branch] gh/bdhirsh/27/base -> origin/gh/bdhirsh/27/base 2022-03-17T19:04:58.1718751Z * [new branch] gh/bdhirsh/27/head -> origin/gh/bdhirsh/27/head 2022-03-17T19:04:58.1721795Z * [new branch] gh/bdhirsh/27/orig -> origin/gh/bdhirsh/27/orig 2022-03-17T19:04:58.1726280Z * [new branch] gh/bdhirsh/29/base -> origin/gh/bdhirsh/29/base 2022-03-17T19:04:58.1729991Z * [new branch] gh/bdhirsh/29/head -> origin/gh/bdhirsh/29/head 2022-03-17T19:04:58.1732997Z * [new branch] gh/bdhirsh/29/orig -> origin/gh/bdhirsh/29/orig 2022-03-17T19:04:58.1736901Z * [new branch] gh/bdhirsh/35/base -> origin/gh/bdhirsh/35/base 2022-03-17T19:04:58.1737998Z * [new branch] gh/bdhirsh/35/head -> origin/gh/bdhirsh/35/head 2022-03-17T19:04:58.1739462Z * [new branch] gh/bdhirsh/35/orig -> origin/gh/bdhirsh/35/orig 2022-03-17T19:04:58.1741275Z * [new branch] gh/bdhirsh/36/base -> origin/gh/bdhirsh/36/base 2022-03-17T19:04:58.1742391Z * [new branch] gh/bdhirsh/36/head -> origin/gh/bdhirsh/36/head 2022-03-17T19:04:58.1743780Z * [new branch] gh/bdhirsh/36/orig -> origin/gh/bdhirsh/36/orig 2022-03-17T19:04:58.1745708Z * [new branch] gh/bdhirsh/37/base -> origin/gh/bdhirsh/37/base 2022-03-17T19:04:58.1746792Z * [new branch] gh/bdhirsh/37/head -> origin/gh/bdhirsh/37/head 2022-03-17T19:04:58.1748159Z * [new branch] gh/bdhirsh/37/orig -> origin/gh/bdhirsh/37/orig 2022-03-17T19:04:58.1749966Z * [new branch] gh/bdhirsh/38/base -> origin/gh/bdhirsh/38/base 2022-03-17T19:04:58.1751101Z * [new branch] gh/bdhirsh/38/head -> origin/gh/bdhirsh/38/head 2022-03-17T19:04:58.1752575Z * [new branch] gh/bdhirsh/38/orig -> origin/gh/bdhirsh/38/orig 2022-03-17T19:04:58.1754459Z * [new branch] gh/bdhirsh/39/base -> origin/gh/bdhirsh/39/base 2022-03-17T19:04:58.1755838Z * [new branch] gh/bdhirsh/39/head -> origin/gh/bdhirsh/39/head 2022-03-17T19:04:58.1757083Z * [new branch] gh/bdhirsh/39/orig -> origin/gh/bdhirsh/39/orig 2022-03-17T19:04:58.1759053Z * [new branch] gh/bdhirsh/41/base -> origin/gh/bdhirsh/41/base 2022-03-17T19:04:58.1761190Z * [new branch] gh/bdhirsh/41/head -> origin/gh/bdhirsh/41/head 2022-03-17T19:04:58.1762270Z * [new branch] gh/bdhirsh/41/orig -> origin/gh/bdhirsh/41/orig 2022-03-17T19:04:58.1764366Z * [new branch] gh/bdhirsh/42/base -> origin/gh/bdhirsh/42/base 2022-03-17T19:04:58.1765463Z * [new branch] gh/bdhirsh/42/head -> origin/gh/bdhirsh/42/head 2022-03-17T19:04:58.1766830Z * [new branch] gh/bdhirsh/42/orig -> origin/gh/bdhirsh/42/orig 2022-03-17T19:04:58.1768361Z * [new branch] gh/bdhirsh/43/base -> origin/gh/bdhirsh/43/base 2022-03-17T19:04:58.1769793Z * [new branch] gh/bdhirsh/43/head -> origin/gh/bdhirsh/43/head 2022-03-17T19:04:58.1771173Z * [new branch] gh/bdhirsh/43/orig -> origin/gh/bdhirsh/43/orig 2022-03-17T19:04:58.1772603Z * [new branch] gh/bdhirsh/44/base -> origin/gh/bdhirsh/44/base 2022-03-17T19:04:58.1773920Z * [new branch] gh/bdhirsh/44/head -> origin/gh/bdhirsh/44/head 2022-03-17T19:04:58.1775247Z * [new branch] gh/bdhirsh/44/orig -> origin/gh/bdhirsh/44/orig 2022-03-17T19:04:58.1776748Z * [new branch] gh/bdhirsh/45/base -> origin/gh/bdhirsh/45/base 2022-03-17T19:04:58.1778123Z * [new branch] gh/bdhirsh/45/head -> origin/gh/bdhirsh/45/head 2022-03-17T19:04:58.1779409Z * [new branch] gh/bdhirsh/45/orig -> origin/gh/bdhirsh/45/orig 2022-03-17T19:04:58.1780974Z * [new branch] gh/bdhirsh/46/base -> origin/gh/bdhirsh/46/base 2022-03-17T19:04:58.1782272Z * [new branch] gh/bdhirsh/46/head -> origin/gh/bdhirsh/46/head 2022-03-17T19:04:58.1783660Z * [new branch] gh/bdhirsh/46/orig -> origin/gh/bdhirsh/46/orig 2022-03-17T19:04:58.1785124Z * [new branch] gh/bdhirsh/47/base -> origin/gh/bdhirsh/47/base 2022-03-17T19:04:58.1786462Z * [new branch] gh/bdhirsh/47/head -> origin/gh/bdhirsh/47/head 2022-03-17T19:04:58.1787816Z * [new branch] gh/bdhirsh/47/orig -> origin/gh/bdhirsh/47/orig 2022-03-17T19:04:58.1789336Z * [new branch] gh/bdhirsh/48/base -> origin/gh/bdhirsh/48/base 2022-03-17T19:04:58.1790701Z * [new branch] gh/bdhirsh/48/head -> origin/gh/bdhirsh/48/head 2022-03-17T19:04:58.1792134Z * [new branch] gh/bdhirsh/48/orig -> origin/gh/bdhirsh/48/orig 2022-03-17T19:04:58.1793935Z * [new branch] gh/bdhirsh/55/base -> origin/gh/bdhirsh/55/base 2022-03-17T19:04:58.1795035Z * [new branch] gh/bdhirsh/55/head -> origin/gh/bdhirsh/55/head 2022-03-17T19:04:58.1797013Z * [new branch] gh/bdhirsh/6/base -> origin/gh/bdhirsh/6/base 2022-03-17T19:04:58.1798105Z * [new branch] gh/bdhirsh/6/head -> origin/gh/bdhirsh/6/head 2022-03-17T19:04:58.1799485Z * [new branch] gh/bdhirsh/6/orig -> origin/gh/bdhirsh/6/orig 2022-03-17T19:04:58.1801364Z * [new branch] gh/bdhirsh/63/base -> origin/gh/bdhirsh/63/base 2022-03-17T19:04:58.1802902Z * [new branch] gh/bdhirsh/63/head -> origin/gh/bdhirsh/63/head 2022-03-17T19:04:58.1804324Z * [new branch] gh/bdhirsh/63/orig -> origin/gh/bdhirsh/63/orig 2022-03-17T19:04:58.1806016Z * [new branch] gh/bdhirsh/71/base -> origin/gh/bdhirsh/71/base 2022-03-17T19:04:58.1807406Z * [new branch] gh/bdhirsh/71/head -> origin/gh/bdhirsh/71/head 2022-03-17T19:04:58.1808934Z * [new branch] gh/bdhirsh/71/orig -> origin/gh/bdhirsh/71/orig 2022-03-17T19:04:58.1810099Z * [new branch] gh/bdhirsh/81/base -> origin/gh/bdhirsh/81/base 2022-03-17T19:04:58.1811623Z * [new branch] gh/bdhirsh/81/head -> origin/gh/bdhirsh/81/head 2022-03-17T19:04:58.1813007Z * [new branch] gh/bdhirsh/81/orig -> origin/gh/bdhirsh/81/orig 2022-03-17T19:04:58.1814558Z * [new branch] gh/bdhirsh/9/base -> origin/gh/bdhirsh/9/base 2022-03-17T19:04:58.1815924Z * [new branch] gh/bdhirsh/9/head -> origin/gh/bdhirsh/9/head 2022-03-17T19:04:58.1817305Z * [new branch] gh/bdhirsh/9/orig -> origin/gh/bdhirsh/9/orig 2022-03-17T19:04:58.1819390Z * [new branch] gh/bdhirsh/96/base -> origin/gh/bdhirsh/96/base 2022-03-17T19:04:58.1821129Z * [new branch] gh/bdhirsh/96/head -> origin/gh/bdhirsh/96/head 2022-03-17T19:04:58.1822508Z * [new branch] gh/bdhirsh/96/orig -> origin/gh/bdhirsh/96/orig 2022-03-17T19:04:58.1824521Z * [new branch] gh/beauby/1/base -> origin/gh/beauby/1/base 2022-03-17T19:04:58.1825895Z * [new branch] gh/beauby/1/head -> origin/gh/beauby/1/head 2022-03-17T19:04:58.1827285Z * [new branch] gh/beauby/2/base -> origin/gh/beauby/2/base 2022-03-17T19:04:58.1828817Z * [new branch] gh/beauby/3/base -> origin/gh/beauby/3/base 2022-03-17T19:04:58.1830741Z * [new branch] gh/bertmaher/172/base -> origin/gh/bertmaher/172/base 2022-03-17T19:04:58.1832162Z * [new branch] gh/bertmaher/172/head -> origin/gh/bertmaher/172/head 2022-03-17T19:04:58.1833451Z * [new branch] gh/bertmaher/172/orig -> origin/gh/bertmaher/172/orig 2022-03-17T19:04:58.1835170Z * [new branch] gh/bertmaher/79/base -> origin/gh/bertmaher/79/base 2022-03-17T19:04:58.1836580Z * [new branch] gh/bertmaher/79/head -> origin/gh/bertmaher/79/head 2022-03-17T19:04:58.1838044Z * [new branch] gh/bertmaher/80/base -> origin/gh/bertmaher/80/base 2022-03-17T19:04:58.1839425Z * [new branch] gh/bertmaher/80/head -> origin/gh/bertmaher/80/head 2022-03-17T19:04:58.1841375Z * [new branch] gh/bhosmer/10/base -> origin/gh/bhosmer/10/base 2022-03-17T19:04:58.1843312Z * [new branch] gh/bhosmer/10/head -> origin/gh/bhosmer/10/head 2022-03-17T19:04:58.1844707Z * [new branch] gh/bhosmer/10/orig -> origin/gh/bhosmer/10/orig 2022-03-17T19:04:58.1846450Z * [new branch] gh/bhosmer/11/base -> origin/gh/bhosmer/11/base 2022-03-17T19:04:58.1847861Z * [new branch] gh/bhosmer/11/head -> origin/gh/bhosmer/11/head 2022-03-17T19:04:58.1849248Z * [new branch] gh/bhosmer/11/orig -> origin/gh/bhosmer/11/orig 2022-03-17T19:04:58.1850748Z * [new branch] gh/bhosmer/12/base -> origin/gh/bhosmer/12/base 2022-03-17T19:04:58.1854267Z * [new branch] gh/bhosmer/12/head -> origin/gh/bhosmer/12/head 2022-03-17T19:04:58.1855585Z * [new branch] gh/bhosmer/12/orig -> origin/gh/bhosmer/12/orig 2022-03-17T19:04:58.1857108Z * [new branch] gh/bhosmer/13/base -> origin/gh/bhosmer/13/base 2022-03-17T19:04:58.1858700Z * [new branch] gh/bhosmer/13/head -> origin/gh/bhosmer/13/head 2022-03-17T19:04:58.1860237Z * [new branch] gh/bhosmer/13/orig -> origin/gh/bhosmer/13/orig 2022-03-17T19:04:58.1861737Z * [new branch] gh/bhosmer/14/base -> origin/gh/bhosmer/14/base 2022-03-17T19:04:58.1863186Z * [new branch] gh/bhosmer/14/head -> origin/gh/bhosmer/14/head 2022-03-17T19:04:58.1864541Z * [new branch] gh/bhosmer/14/orig -> origin/gh/bhosmer/14/orig 2022-03-17T19:04:58.1866573Z * [new branch] gh/bhosmer/17/base -> origin/gh/bhosmer/17/base 2022-03-17T19:04:58.1867930Z * [new branch] gh/bhosmer/17/head -> origin/gh/bhosmer/17/head 2022-03-17T19:04:58.1869364Z * [new branch] gh/bhosmer/17/orig -> origin/gh/bhosmer/17/orig 2022-03-17T19:04:58.1870886Z * [new branch] gh/bhosmer/18/base -> origin/gh/bhosmer/18/base 2022-03-17T19:04:58.1872314Z * [new branch] gh/bhosmer/18/head -> origin/gh/bhosmer/18/head 2022-03-17T19:04:58.1873694Z * [new branch] gh/bhosmer/18/orig -> origin/gh/bhosmer/18/orig 2022-03-17T19:04:58.1875232Z * [new branch] gh/bhosmer/19/base -> origin/gh/bhosmer/19/base 2022-03-17T19:04:58.1876570Z * [new branch] gh/bhosmer/19/head -> origin/gh/bhosmer/19/head 2022-03-17T19:04:58.1877875Z * [new branch] gh/bhosmer/19/orig -> origin/gh/bhosmer/19/orig 2022-03-17T19:04:58.1879928Z * [new branch] gh/bhosmer/2/base -> origin/gh/bhosmer/2/base 2022-03-17T19:04:58.1881433Z * [new branch] gh/bhosmer/2/head -> origin/gh/bhosmer/2/head 2022-03-17T19:04:58.1883227Z * [new branch] gh/bhosmer/2/orig -> origin/gh/bhosmer/2/orig 2022-03-17T19:04:58.1884977Z * [new branch] gh/bhosmer/21/base -> origin/gh/bhosmer/21/base 2022-03-17T19:04:58.1886357Z * [new branch] gh/bhosmer/21/head -> origin/gh/bhosmer/21/head 2022-03-17T19:04:58.1887671Z * [new branch] gh/bhosmer/21/orig -> origin/gh/bhosmer/21/orig 2022-03-17T19:04:58.1889457Z * [new branch] gh/bhosmer/24/base -> origin/gh/bhosmer/24/base 2022-03-17T19:04:58.1890749Z * [new branch] gh/bhosmer/24/head -> origin/gh/bhosmer/24/head 2022-03-17T19:04:58.1892026Z * [new branch] gh/bhosmer/24/orig -> origin/gh/bhosmer/24/orig 2022-03-17T19:04:58.1893569Z * [new branch] gh/bhosmer/25/base -> origin/gh/bhosmer/25/base 2022-03-17T19:04:58.1895040Z * [new branch] gh/bhosmer/25/head -> origin/gh/bhosmer/25/head 2022-03-17T19:04:58.1896470Z * [new branch] gh/bhosmer/25/orig -> origin/gh/bhosmer/25/orig 2022-03-17T19:04:58.1897882Z * [new branch] gh/bhosmer/3/base -> origin/gh/bhosmer/3/base 2022-03-17T19:04:58.1899759Z * [new branch] gh/bhosmer/3/head -> origin/gh/bhosmer/3/head 2022-03-17T19:04:58.1901050Z * [new branch] gh/bhosmer/3/orig -> origin/gh/bhosmer/3/orig 2022-03-17T19:04:58.1902795Z * [new branch] gh/bhosmer/33/base -> origin/gh/bhosmer/33/base 2022-03-17T19:04:58.1904123Z * [new branch] gh/bhosmer/33/head -> origin/gh/bhosmer/33/head 2022-03-17T19:04:58.1905406Z * [new branch] gh/bhosmer/33/orig -> origin/gh/bhosmer/33/orig 2022-03-17T19:04:58.1907350Z * [new branch] gh/bhosmer/42/base -> origin/gh/bhosmer/42/base 2022-03-17T19:04:58.1908409Z * [new branch] gh/bhosmer/42/head -> origin/gh/bhosmer/42/head 2022-03-17T19:04:58.1909890Z * [new branch] gh/bhosmer/42/orig -> origin/gh/bhosmer/42/orig 2022-03-17T19:04:58.1911738Z * [new branch] gh/bhosmer/43/base -> origin/gh/bhosmer/43/base 2022-03-17T19:04:58.1912969Z * [new branch] gh/bhosmer/43/head -> origin/gh/bhosmer/43/head 2022-03-17T19:04:58.1914288Z * [new branch] gh/bhosmer/43/orig -> origin/gh/bhosmer/43/orig 2022-03-17T19:04:58.1916222Z * [new branch] gh/bhosmer/6/base -> origin/gh/bhosmer/6/base 2022-03-17T19:04:58.1917731Z * [new branch] gh/bhosmer/6/head -> origin/gh/bhosmer/6/head 2022-03-17T19:04:58.1919324Z * [new branch] gh/bhosmer/6/orig -> origin/gh/bhosmer/6/orig 2022-03-17T19:04:58.1920790Z * [new branch] gh/bhosmer/7/base -> origin/gh/bhosmer/7/base 2022-03-17T19:04:58.1922220Z * [new branch] gh/bhosmer/7/head -> origin/gh/bhosmer/7/head 2022-03-17T19:04:58.1923541Z * [new branch] gh/bhosmer/7/orig -> origin/gh/bhosmer/7/orig 2022-03-17T19:04:58.1925432Z * [new branch] gh/bhosmer/8/base -> origin/gh/bhosmer/8/base 2022-03-17T19:04:58.1926803Z * [new branch] gh/bhosmer/8/head -> origin/gh/bhosmer/8/head 2022-03-17T19:04:58.1928625Z * [new branch] gh/bhosmer/8/orig -> origin/gh/bhosmer/8/orig 2022-03-17T19:04:58.1930430Z * [new branch] gh/bhosmer/9/base -> origin/gh/bhosmer/9/base 2022-03-17T19:04:58.1931774Z * [new branch] gh/bhosmer/9/head -> origin/gh/bhosmer/9/head 2022-03-17T19:04:58.1933372Z * [new branch] gh/bhosmer/9/orig -> origin/gh/bhosmer/9/orig 2022-03-17T19:04:58.1935814Z * [new branch] gh/blefaudeux/4/base -> origin/gh/blefaudeux/4/base 2022-03-17T19:04:58.1937249Z * [new branch] gh/blefaudeux/4/head -> origin/gh/blefaudeux/4/head 2022-03-17T19:04:58.1938831Z * [new branch] gh/blefaudeux/4/orig -> origin/gh/blefaudeux/4/orig 2022-03-17T19:04:58.1940763Z * [new branch] gh/bowangbj/1/base -> origin/gh/bowangbj/1/base 2022-03-17T19:04:58.1941909Z * [new branch] gh/bowangbj/1/orig -> origin/gh/bowangbj/1/orig 2022-03-17T19:04:58.1944002Z * [new branch] gh/bowangbj/11/base -> origin/gh/bowangbj/11/base 2022-03-17T19:04:58.1945322Z * [new branch] gh/bowangbj/11/head -> origin/gh/bowangbj/11/head 2022-03-17T19:04:58.1946732Z * [new branch] gh/bowangbj/11/orig -> origin/gh/bowangbj/11/orig 2022-03-17T19:04:58.1948560Z * [new branch] gh/bowangbj/13/base -> origin/gh/bowangbj/13/base 2022-03-17T19:04:58.1949720Z * [new branch] gh/bowangbj/13/head -> origin/gh/bowangbj/13/head 2022-03-17T19:04:58.1951181Z * [new branch] gh/bowangbj/13/orig -> origin/gh/bowangbj/13/orig 2022-03-17T19:04:58.1953009Z * [new branch] gh/bowangbj/2/base -> origin/gh/bowangbj/2/base 2022-03-17T19:04:58.1954168Z * [new branch] gh/bowangbj/2/orig -> origin/gh/bowangbj/2/orig 2022-03-17T19:04:58.1956258Z * [new branch] gh/bowangbj/3/base -> origin/gh/bowangbj/3/base 2022-03-17T19:04:58.1957616Z * [new branch] gh/bowangbj/3/head -> origin/gh/bowangbj/3/head 2022-03-17T19:04:58.1958970Z * [new branch] gh/bowangbj/3/orig -> origin/gh/bowangbj/3/orig 2022-03-17T19:04:58.1960829Z * [new branch] gh/bowangbj/4/base -> origin/gh/bowangbj/4/base 2022-03-17T19:04:58.1962039Z * [new branch] gh/bowangbj/4/orig -> origin/gh/bowangbj/4/orig 2022-03-17T19:04:58.1964199Z * [new branch] gh/bowangbj/9/base -> origin/gh/bowangbj/9/base 2022-03-17T19:04:58.1965364Z * [new branch] gh/bowangbj/9/head -> origin/gh/bowangbj/9/head 2022-03-17T19:04:58.1966979Z * [new branch] gh/bowangbj/9/orig -> origin/gh/bowangbj/9/orig 2022-03-17T19:04:58.1969362Z * [new branch] gh/bwasti/10/base -> origin/gh/bwasti/10/base 2022-03-17T19:04:58.1970847Z * [new branch] gh/bwasti/10/head -> origin/gh/bwasti/10/head 2022-03-17T19:04:58.1973050Z * [new branch] gh/bwasti/13/base -> origin/gh/bwasti/13/base 2022-03-17T19:04:58.1974426Z * [new branch] gh/bwasti/13/head -> origin/gh/bwasti/13/head 2022-03-17T19:04:58.1976361Z * [new branch] gh/bwasti/14/base -> origin/gh/bwasti/14/base 2022-03-17T19:04:58.1977811Z * [new branch] gh/bwasti/14/head -> origin/gh/bwasti/14/head 2022-03-17T19:04:58.1979266Z * [new branch] gh/bwasti/16/base -> origin/gh/bwasti/16/base 2022-03-17T19:04:58.1980674Z * [new branch] gh/bwasti/16/head -> origin/gh/bwasti/16/head 2022-03-17T19:04:58.1982218Z * [new branch] gh/bwasti/17/base -> origin/gh/bwasti/17/base 2022-03-17T19:04:58.1983490Z * [new branch] gh/bwasti/17/head -> origin/gh/bwasti/17/head 2022-03-17T19:04:58.1985138Z * [new branch] gh/bwasti/25/base -> origin/gh/bwasti/25/base 2022-03-17T19:04:58.1986561Z * [new branch] gh/bwasti/25/head -> origin/gh/bwasti/25/head 2022-03-17T19:04:58.1988266Z * [new branch] gh/bwasti/26/base -> origin/gh/bwasti/26/base 2022-03-17T19:04:58.1989626Z * [new branch] gh/bwasti/26/head -> origin/gh/bwasti/26/head 2022-03-17T19:04:58.1991367Z * [new branch] gh/bwasti/28/base -> origin/gh/bwasti/28/base 2022-03-17T19:04:58.1992625Z * [new branch] gh/bwasti/28/head -> origin/gh/bwasti/28/head 2022-03-17T19:04:58.1994136Z * [new branch] gh/bwasti/31/base -> origin/gh/bwasti/31/base 2022-03-17T19:04:58.1995553Z * [new branch] gh/bwasti/31/head -> origin/gh/bwasti/31/head 2022-03-17T19:04:58.1996898Z * [new branch] gh/bwasti/31/orig -> origin/gh/bwasti/31/orig 2022-03-17T19:04:58.1998692Z * [new branch] gh/bwasti/33/base -> origin/gh/bwasti/33/base 2022-03-17T19:04:58.2000135Z * [new branch] gh/bwasti/33/head -> origin/gh/bwasti/33/head 2022-03-17T19:04:58.2001485Z * [new branch] gh/bwasti/33/orig -> origin/gh/bwasti/33/orig 2022-03-17T19:04:58.2003900Z * [new branch] gh/bwasti/35/base -> origin/gh/bwasti/35/base 2022-03-17T19:04:58.2005143Z * [new branch] gh/bwasti/35/head -> origin/gh/bwasti/35/head 2022-03-17T19:04:58.2006507Z * [new branch] gh/bwasti/35/orig -> origin/gh/bwasti/35/orig 2022-03-17T19:04:58.2007957Z * [new branch] gh/bwasti/36/base -> origin/gh/bwasti/36/base 2022-03-17T19:04:58.2009295Z * [new branch] gh/bwasti/36/head -> origin/gh/bwasti/36/head 2022-03-17T19:04:58.2010618Z * [new branch] gh/bwasti/36/orig -> origin/gh/bwasti/36/orig 2022-03-17T19:04:58.2012258Z * [new branch] gh/bwasti/40/base -> origin/gh/bwasti/40/base 2022-03-17T19:04:58.2013584Z * [new branch] gh/bwasti/40/head -> origin/gh/bwasti/40/head 2022-03-17T19:04:58.2015054Z * [new branch] gh/bwasti/41/base -> origin/gh/bwasti/41/base 2022-03-17T19:04:58.2016427Z * [new branch] gh/bwasti/41/head -> origin/gh/bwasti/41/head 2022-03-17T19:04:58.2017803Z * [new branch] gh/bwasti/41/orig -> origin/gh/bwasti/41/orig 2022-03-17T19:04:58.2019497Z * [new branch] gh/bwasti/49/base -> origin/gh/bwasti/49/base 2022-03-17T19:04:58.2020912Z * [new branch] gh/bwasti/49/head -> origin/gh/bwasti/49/head 2022-03-17T19:04:58.2022257Z * [new branch] gh/bwasti/49/orig -> origin/gh/bwasti/49/orig 2022-03-17T19:04:58.2023902Z * [new branch] gh/bwasti/51/base -> origin/gh/bwasti/51/base 2022-03-17T19:04:58.2025237Z * [new branch] gh/bwasti/51/head -> origin/gh/bwasti/51/head 2022-03-17T19:04:58.2026564Z * [new branch] gh/bwasti/51/orig -> origin/gh/bwasti/51/orig 2022-03-17T19:04:58.2028237Z * [new branch] gh/bwasti/9/base -> origin/gh/bwasti/9/base 2022-03-17T19:04:58.2029431Z * [new branch] gh/bwasti/9/head -> origin/gh/bwasti/9/head 2022-03-17T19:04:58.2031524Z * [new branch] gh/bzinodev/1/base -> origin/gh/bzinodev/1/base 2022-03-17T19:04:58.2032958Z * [new branch] gh/bzinodev/1/head -> origin/gh/bzinodev/1/head 2022-03-17T19:04:58.2034418Z * [new branch] gh/bzinodev/2/base -> origin/gh/bzinodev/2/base 2022-03-17T19:04:58.2035797Z * [new branch] gh/bzinodev/2/head -> origin/gh/bzinodev/2/head 2022-03-17T19:04:58.2037611Z * [new branch] gh/bzinodev/20/base -> origin/gh/bzinodev/20/base 2022-03-17T19:04:58.2038971Z * [new branch] gh/bzinodev/20/head -> origin/gh/bzinodev/20/head 2022-03-17T19:04:58.2040256Z * [new branch] gh/bzinodev/20/orig -> origin/gh/bzinodev/20/orig 2022-03-17T19:04:58.2042041Z * [new branch] gh/bzinodev/6/base -> origin/gh/bzinodev/6/base 2022-03-17T19:04:58.2043805Z * [new branch] gh/bzinodev/6/head -> origin/gh/bzinodev/6/head 2022-03-17T19:04:58.2045118Z * [new branch] gh/bzinodev/6/orig -> origin/gh/bzinodev/6/orig 2022-03-17T19:04:58.2046790Z * [new branch] gh/bzinodev/7/base -> origin/gh/bzinodev/7/base 2022-03-17T19:04:58.2048182Z * [new branch] gh/bzinodev/7/head -> origin/gh/bzinodev/7/head 2022-03-17T19:04:58.2049578Z * [new branch] gh/bzinodev/7/orig -> origin/gh/bzinodev/7/orig 2022-03-17T19:04:58.2051804Z * [new branch] gh/cbalioglu/24/base -> origin/gh/cbalioglu/24/base 2022-03-17T19:04:58.2053223Z * [new branch] gh/cbalioglu/24/orig -> origin/gh/cbalioglu/24/orig 2022-03-17T19:04:58.2054953Z * [new branch] gh/cbalioglu/25/base -> origin/gh/cbalioglu/25/base 2022-03-17T19:04:58.2056323Z * [new branch] gh/cbalioglu/25/orig -> origin/gh/cbalioglu/25/orig 2022-03-17T19:04:58.2057994Z * [new branch] gh/cbalioglu/30/base -> origin/gh/cbalioglu/30/base 2022-03-17T19:04:58.2059374Z * [new branch] gh/cbalioglu/30/orig -> origin/gh/cbalioglu/30/orig 2022-03-17T19:04:58.2061176Z * [new branch] gh/cbalioglu/31/base -> origin/gh/cbalioglu/31/base 2022-03-17T19:04:58.2062609Z * [new branch] gh/cbalioglu/31/orig -> origin/gh/cbalioglu/31/orig 2022-03-17T19:04:58.2064377Z * [new branch] gh/cbalioglu/37/base -> origin/gh/cbalioglu/37/base 2022-03-17T19:04:58.2065623Z * [new branch] gh/cbalioglu/37/head -> origin/gh/cbalioglu/37/head 2022-03-17T19:04:58.2067123Z * [new branch] gh/cbalioglu/38/base -> origin/gh/cbalioglu/38/base 2022-03-17T19:04:58.2068442Z * [new branch] gh/cbalioglu/38/head -> origin/gh/cbalioglu/38/head 2022-03-17T19:04:58.2073372Z * [new branch] gh/cbalioglu/39/base -> origin/gh/cbalioglu/39/base 2022-03-17T19:04:58.2073719Z * [new branch] gh/cbalioglu/39/head -> origin/gh/cbalioglu/39/head 2022-03-17T19:04:58.2074087Z * [new branch] gh/cbalioglu/40/base -> origin/gh/cbalioglu/40/base 2022-03-17T19:04:58.2074546Z * [new branch] gh/cbalioglu/40/head -> origin/gh/cbalioglu/40/head 2022-03-17T19:04:58.2077390Z * [new branch] gh/cbalioglu/46/base -> origin/gh/cbalioglu/46/base 2022-03-17T19:04:58.2078435Z * [new branch] gh/cbalioglu/46/orig -> origin/gh/cbalioglu/46/orig 2022-03-17T19:04:58.2079676Z * [new branch] gh/cbalioglu/48/base -> origin/gh/cbalioglu/48/base 2022-03-17T19:04:58.2080768Z * [new branch] gh/cbalioglu/48/orig -> origin/gh/cbalioglu/48/orig 2022-03-17T19:04:58.2082880Z * [new branch] gh/cbalioglu/49/base -> origin/gh/cbalioglu/49/base 2022-03-17T19:04:58.2084244Z * [new branch] gh/cbalioglu/49/orig -> origin/gh/cbalioglu/49/orig 2022-03-17T19:04:58.2085891Z * [new branch] gh/cbalioglu/56/base -> origin/gh/cbalioglu/56/base 2022-03-17T19:04:58.2087337Z * [new branch] gh/cbalioglu/56/orig -> origin/gh/cbalioglu/56/orig 2022-03-17T19:04:58.2088991Z * [new branch] gh/cbalioglu/57/base -> origin/gh/cbalioglu/57/base 2022-03-17T19:04:58.2090823Z * [new branch] gh/cbalioglu/57/orig -> origin/gh/cbalioglu/57/orig 2022-03-17T19:04:58.2092154Z * [new branch] gh/cbalioglu/58/base -> origin/gh/cbalioglu/58/base 2022-03-17T19:04:58.2093491Z * [new branch] gh/cbalioglu/58/orig -> origin/gh/cbalioglu/58/orig 2022-03-17T19:04:58.2095642Z * [new branch] gh/cccclai/10/base -> origin/gh/cccclai/10/base 2022-03-17T19:04:58.2097087Z * [new branch] gh/cccclai/10/head -> origin/gh/cccclai/10/head 2022-03-17T19:04:58.2098307Z * [new branch] gh/cccclai/10/orig -> origin/gh/cccclai/10/orig 2022-03-17T19:04:58.2099940Z * [new branch] gh/cccclai/101/base -> origin/gh/cccclai/101/base 2022-03-17T19:04:58.2101268Z * [new branch] gh/cccclai/101/head -> origin/gh/cccclai/101/head 2022-03-17T19:04:58.2102692Z * [new branch] gh/cccclai/101/orig -> origin/gh/cccclai/101/orig 2022-03-17T19:04:58.2104387Z * [new branch] gh/cccclai/104/base -> origin/gh/cccclai/104/base 2022-03-17T19:04:58.2105786Z * [new branch] gh/cccclai/104/head -> origin/gh/cccclai/104/head 2022-03-17T19:04:58.2107056Z * [new branch] gh/cccclai/104/orig -> origin/gh/cccclai/104/orig 2022-03-17T19:04:58.2108816Z * [new branch] gh/cccclai/106/base -> origin/gh/cccclai/106/base 2022-03-17T19:04:58.2110119Z * [new branch] gh/cccclai/106/head -> origin/gh/cccclai/106/head 2022-03-17T19:04:58.2111405Z * [new branch] gh/cccclai/106/orig -> origin/gh/cccclai/106/orig 2022-03-17T19:04:58.2113227Z * [new branch] gh/cccclai/109/base -> origin/gh/cccclai/109/base 2022-03-17T19:04:58.2114873Z * [new branch] gh/cccclai/109/head -> origin/gh/cccclai/109/head 2022-03-17T19:04:58.2116267Z * [new branch] gh/cccclai/109/orig -> origin/gh/cccclai/109/orig 2022-03-17T19:04:58.2118061Z * [new branch] gh/cccclai/110/base -> origin/gh/cccclai/110/base 2022-03-17T19:04:58.2119383Z * [new branch] gh/cccclai/110/head -> origin/gh/cccclai/110/head 2022-03-17T19:04:58.2120767Z * [new branch] gh/cccclai/110/orig -> origin/gh/cccclai/110/orig 2022-03-17T19:04:58.2123236Z * [new branch] gh/cccclai/111/base -> origin/gh/cccclai/111/base 2022-03-17T19:04:58.2124278Z * [new branch] gh/cccclai/111/head -> origin/gh/cccclai/111/head 2022-03-17T19:04:58.2125507Z * [new branch] gh/cccclai/111/orig -> origin/gh/cccclai/111/orig 2022-03-17T19:04:58.2127522Z * [new branch] gh/cccclai/112/base -> origin/gh/cccclai/112/base 2022-03-17T19:04:58.2128659Z * [new branch] gh/cccclai/112/head -> origin/gh/cccclai/112/head 2022-03-17T19:04:58.2130012Z * [new branch] gh/cccclai/112/orig -> origin/gh/cccclai/112/orig 2022-03-17T19:04:58.2132091Z * [new branch] gh/cccclai/113/base -> origin/gh/cccclai/113/base 2022-03-17T19:04:58.2133414Z * [new branch] gh/cccclai/113/head -> origin/gh/cccclai/113/head 2022-03-17T19:04:58.2134785Z * [new branch] gh/cccclai/113/orig -> origin/gh/cccclai/113/orig 2022-03-17T19:04:58.2136728Z * [new branch] gh/cccclai/116/base -> origin/gh/cccclai/116/base 2022-03-17T19:04:58.2138019Z * [new branch] gh/cccclai/116/head -> origin/gh/cccclai/116/head 2022-03-17T19:04:58.2139487Z * [new branch] gh/cccclai/116/orig -> origin/gh/cccclai/116/orig 2022-03-17T19:04:58.2141183Z * [new branch] gh/cccclai/121/base -> origin/gh/cccclai/121/base 2022-03-17T19:04:58.2142477Z * [new branch] gh/cccclai/121/head -> origin/gh/cccclai/121/head 2022-03-17T19:04:58.2143813Z * [new branch] gh/cccclai/121/orig -> origin/gh/cccclai/121/orig 2022-03-17T19:04:58.2146409Z * [new branch] gh/cccclai/122/base -> origin/gh/cccclai/122/base 2022-03-17T19:04:58.2147429Z * [new branch] gh/cccclai/122/head -> origin/gh/cccclai/122/head 2022-03-17T19:04:58.2148788Z * [new branch] gh/cccclai/122/orig -> origin/gh/cccclai/122/orig 2022-03-17T19:04:58.2150753Z * [new branch] gh/cccclai/128/base -> origin/gh/cccclai/128/base 2022-03-17T19:04:58.2151944Z * [new branch] gh/cccclai/128/head -> origin/gh/cccclai/128/head 2022-03-17T19:04:58.2153258Z * [new branch] gh/cccclai/128/orig -> origin/gh/cccclai/128/orig 2022-03-17T19:04:58.2155398Z * [new branch] gh/cccclai/130/base -> origin/gh/cccclai/130/base 2022-03-17T19:04:58.2156610Z * [new branch] gh/cccclai/130/head -> origin/gh/cccclai/130/head 2022-03-17T19:04:58.2158155Z * [new branch] gh/cccclai/130/orig -> origin/gh/cccclai/130/orig 2022-03-17T19:04:58.2159869Z * [new branch] gh/cccclai/136/base -> origin/gh/cccclai/136/base 2022-03-17T19:04:58.2161316Z * [new branch] gh/cccclai/136/head -> origin/gh/cccclai/136/head 2022-03-17T19:04:58.2163427Z * [new branch] gh/cccclai/136/orig -> origin/gh/cccclai/136/orig 2022-03-17T19:04:58.2165572Z * [new branch] gh/cccclai/139/base -> origin/gh/cccclai/139/base 2022-03-17T19:04:58.2166549Z * [new branch] gh/cccclai/139/head -> origin/gh/cccclai/139/head 2022-03-17T19:04:58.2167895Z * [new branch] gh/cccclai/139/orig -> origin/gh/cccclai/139/orig 2022-03-17T19:04:58.2169652Z * [new branch] gh/cccclai/14/base -> origin/gh/cccclai/14/base 2022-03-17T19:04:58.2170517Z * [new branch] gh/cccclai/14/head -> origin/gh/cccclai/14/head 2022-03-17T19:04:58.2172817Z * [new branch] gh/cccclai/141/base -> origin/gh/cccclai/141/base 2022-03-17T19:04:58.2173835Z * [new branch] gh/cccclai/141/head -> origin/gh/cccclai/141/head 2022-03-17T19:04:58.2175299Z * [new branch] gh/cccclai/141/orig -> origin/gh/cccclai/141/orig 2022-03-17T19:04:58.2176994Z * [new branch] gh/cccclai/146/base -> origin/gh/cccclai/146/base 2022-03-17T19:04:58.2178451Z * [new branch] gh/cccclai/146/head -> origin/gh/cccclai/146/head 2022-03-17T19:04:58.2179958Z * [new branch] gh/cccclai/147/base -> origin/gh/cccclai/147/base 2022-03-17T19:04:58.2181047Z * [new branch] gh/cccclai/147/head -> origin/gh/cccclai/147/head 2022-03-17T19:04:58.2182875Z * [new branch] gh/cccclai/149/base -> origin/gh/cccclai/149/base 2022-03-17T19:04:58.2184510Z * [new branch] gh/cccclai/149/head -> origin/gh/cccclai/149/head 2022-03-17T19:04:58.2186113Z * [new branch] gh/cccclai/149/orig -> origin/gh/cccclai/149/orig 2022-03-17T19:04:58.2187791Z * [new branch] gh/cccclai/15/base -> origin/gh/cccclai/15/base 2022-03-17T19:04:58.2189816Z * [new branch] gh/cccclai/15/head -> origin/gh/cccclai/15/head 2022-03-17T19:04:58.2190998Z * [new branch] gh/cccclai/15/orig -> origin/gh/cccclai/15/orig 2022-03-17T19:04:58.2193078Z * [new branch] gh/cccclai/154/base -> origin/gh/cccclai/154/base 2022-03-17T19:04:58.2194219Z * [new branch] gh/cccclai/154/head -> origin/gh/cccclai/154/head 2022-03-17T19:04:58.2195845Z * [new branch] gh/cccclai/155/base -> origin/gh/cccclai/155/base 2022-03-17T19:04:58.2196977Z * [new branch] gh/cccclai/155/head -> origin/gh/cccclai/155/head 2022-03-17T19:04:58.2198356Z * [new branch] gh/cccclai/155/orig -> origin/gh/cccclai/155/orig 2022-03-17T19:04:58.2200131Z * [new branch] gh/cccclai/156/base -> origin/gh/cccclai/156/base 2022-03-17T19:04:58.2201732Z * [new branch] gh/cccclai/156/head -> origin/gh/cccclai/156/head 2022-03-17T19:04:58.2202996Z * [new branch] gh/cccclai/156/orig -> origin/gh/cccclai/156/orig 2022-03-17T19:04:58.2204841Z * [new branch] gh/cccclai/157/base -> origin/gh/cccclai/157/base 2022-03-17T19:04:58.2206227Z * [new branch] gh/cccclai/157/head -> origin/gh/cccclai/157/head 2022-03-17T19:04:58.2207591Z * [new branch] gh/cccclai/157/orig -> origin/gh/cccclai/157/orig 2022-03-17T19:04:58.2209098Z * [new branch] gh/cccclai/160/base -> origin/gh/cccclai/160/base 2022-03-17T19:04:58.2210476Z * [new branch] gh/cccclai/160/head -> origin/gh/cccclai/160/head 2022-03-17T19:04:58.2212127Z * [new branch] gh/cccclai/20/base -> origin/gh/cccclai/20/base 2022-03-17T19:04:58.2213503Z * [new branch] gh/cccclai/20/head -> origin/gh/cccclai/20/head 2022-03-17T19:04:58.2214888Z * [new branch] gh/cccclai/20/orig -> origin/gh/cccclai/20/orig 2022-03-17T19:04:58.2216597Z * [new branch] gh/cccclai/26/base -> origin/gh/cccclai/26/base 2022-03-17T19:04:58.2217983Z * [new branch] gh/cccclai/26/head -> origin/gh/cccclai/26/head 2022-03-17T19:04:58.2219174Z * [new branch] gh/cccclai/26/orig -> origin/gh/cccclai/26/orig 2022-03-17T19:04:58.2220981Z * [new branch] gh/cccclai/29/base -> origin/gh/cccclai/29/base 2022-03-17T19:04:58.2222400Z * [new branch] gh/cccclai/29/head -> origin/gh/cccclai/29/head 2022-03-17T19:04:58.2223597Z * [new branch] gh/cccclai/29/orig -> origin/gh/cccclai/29/orig 2022-03-17T19:04:58.2225407Z * [new branch] gh/cccclai/32/base -> origin/gh/cccclai/32/base 2022-03-17T19:04:58.2226809Z * [new branch] gh/cccclai/32/head -> origin/gh/cccclai/32/head 2022-03-17T19:04:58.2228494Z * [new branch] gh/cccclai/33/base -> origin/gh/cccclai/33/base 2022-03-17T19:04:58.2229863Z * [new branch] gh/cccclai/33/head -> origin/gh/cccclai/33/head 2022-03-17T19:04:58.2231562Z * [new branch] gh/cccclai/38/base -> origin/gh/cccclai/38/base 2022-03-17T19:04:58.2233046Z * [new branch] gh/cccclai/38/head -> origin/gh/cccclai/38/head 2022-03-17T19:04:58.2234467Z * [new branch] gh/cccclai/38/orig -> origin/gh/cccclai/38/orig 2022-03-17T19:04:58.2236167Z * [new branch] gh/cccclai/40/base -> origin/gh/cccclai/40/base 2022-03-17T19:04:58.2238403Z * [new branch] gh/cccclai/40/head -> origin/gh/cccclai/40/head 2022-03-17T19:04:58.2243520Z * [new branch] gh/cccclai/40/orig -> origin/gh/cccclai/40/orig 2022-03-17T19:04:58.2244118Z * [new branch] gh/cccclai/46/base -> origin/gh/cccclai/46/base 2022-03-17T19:04:58.2244655Z * [new branch] gh/cccclai/46/head -> origin/gh/cccclai/46/head 2022-03-17T19:04:58.2245156Z * [new branch] gh/cccclai/49/base -> origin/gh/cccclai/49/base 2022-03-17T19:04:58.2246875Z * [new branch] gh/cccclai/49/head -> origin/gh/cccclai/49/head 2022-03-17T19:04:58.2248221Z * [new branch] gh/cccclai/49/orig -> origin/gh/cccclai/49/orig 2022-03-17T19:04:58.2249309Z * [new branch] gh/cccclai/65/base -> origin/gh/cccclai/65/base 2022-03-17T19:04:58.2250595Z * [new branch] gh/cccclai/65/head -> origin/gh/cccclai/65/head 2022-03-17T19:04:58.2251885Z * [new branch] gh/cccclai/65/orig -> origin/gh/cccclai/65/orig 2022-03-17T19:04:58.2254461Z * [new branch] gh/cccclai/71/base -> origin/gh/cccclai/71/base 2022-03-17T19:04:58.2255204Z * [new branch] gh/cccclai/71/head -> origin/gh/cccclai/71/head 2022-03-17T19:04:58.2256656Z * [new branch] gh/cccclai/71/orig -> origin/gh/cccclai/71/orig 2022-03-17T19:04:58.2258246Z * [new branch] gh/cccclai/72/base -> origin/gh/cccclai/72/base 2022-03-17T19:04:58.2280261Z * [new branch] gh/cccclai/72/head -> origin/gh/cccclai/72/head 2022-03-17T19:04:58.2281210Z * [new branch] gh/cccclai/72/orig -> origin/gh/cccclai/72/orig 2022-03-17T19:04:58.2282475Z * [new branch] gh/cccclai/78/base -> origin/gh/cccclai/78/base 2022-03-17T19:04:58.2283459Z * [new branch] gh/cccclai/78/head -> origin/gh/cccclai/78/head 2022-03-17T19:04:58.2284201Z * [new branch] gh/cccclai/78/orig -> origin/gh/cccclai/78/orig 2022-03-17T19:04:58.2284927Z * [new branch] gh/cccclai/83/base -> origin/gh/cccclai/83/base 2022-03-17T19:04:58.2285644Z * [new branch] gh/cccclai/83/head -> origin/gh/cccclai/83/head 2022-03-17T19:04:58.2286367Z * [new branch] gh/cccclai/83/orig -> origin/gh/cccclai/83/orig 2022-03-17T19:04:58.2287465Z * [new branch] gh/cccclai/85/base -> origin/gh/cccclai/85/base 2022-03-17T19:04:58.2288207Z * [new branch] gh/cccclai/85/head -> origin/gh/cccclai/85/head 2022-03-17T19:04:58.2288930Z * [new branch] gh/cccclai/85/orig -> origin/gh/cccclai/85/orig 2022-03-17T19:04:58.2289646Z * [new branch] gh/cccclai/86/base -> origin/gh/cccclai/86/base 2022-03-17T19:04:58.2290356Z * [new branch] gh/cccclai/86/head -> origin/gh/cccclai/86/head 2022-03-17T19:04:58.2291078Z * [new branch] gh/cccclai/86/orig -> origin/gh/cccclai/86/orig 2022-03-17T19:04:58.2291805Z * [new branch] gh/cccclai/87/base -> origin/gh/cccclai/87/base 2022-03-17T19:04:58.2292524Z * [new branch] gh/cccclai/87/head -> origin/gh/cccclai/87/head 2022-03-17T19:04:58.2293244Z * [new branch] gh/cccclai/87/orig -> origin/gh/cccclai/87/orig 2022-03-17T19:04:58.2293991Z * [new branch] gh/cccclai/89/base -> origin/gh/cccclai/89/base 2022-03-17T19:04:58.2294695Z * [new branch] gh/cccclai/89/head -> origin/gh/cccclai/89/head 2022-03-17T19:04:58.2295418Z * [new branch] gh/cccclai/89/orig -> origin/gh/cccclai/89/orig 2022-03-17T19:04:58.2296163Z * [new branch] gh/cccclai/92/base -> origin/gh/cccclai/92/base 2022-03-17T19:04:58.2296882Z * [new branch] gh/cccclai/92/head -> origin/gh/cccclai/92/head 2022-03-17T19:04:58.2297601Z * [new branch] gh/cccclai/92/orig -> origin/gh/cccclai/92/orig 2022-03-17T19:04:58.2298322Z * [new branch] gh/cccclai/93/base -> origin/gh/cccclai/93/base 2022-03-17T19:04:58.2299031Z * [new branch] gh/cccclai/93/head -> origin/gh/cccclai/93/head 2022-03-17T19:04:58.2299750Z * [new branch] gh/cccclai/94/base -> origin/gh/cccclai/94/base 2022-03-17T19:04:58.2300502Z * [new branch] gh/cccclai/94/head -> origin/gh/cccclai/94/head 2022-03-17T19:04:58.2301216Z * [new branch] gh/cccclai/94/orig -> origin/gh/cccclai/94/orig 2022-03-17T19:04:58.2301940Z * [new branch] gh/cccclai/95/base -> origin/gh/cccclai/95/base 2022-03-17T19:04:58.2302693Z * [new branch] gh/cccclai/95/head -> origin/gh/cccclai/95/head 2022-03-17T19:04:58.2303430Z * [new branch] gh/cccclai/95/orig -> origin/gh/cccclai/95/orig 2022-03-17T19:04:58.2304446Z * [new branch] gh/cccclai/99/base -> origin/gh/cccclai/99/base 2022-03-17T19:04:58.2307109Z * [new branch] gh/cccclai/99/head -> origin/gh/cccclai/99/head 2022-03-17T19:04:58.2308406Z * [new branch] gh/cccclai/99/orig -> origin/gh/cccclai/99/orig 2022-03-17T19:04:58.2309945Z * [new branch] gh/changjishi/2/base -> origin/gh/changjishi/2/base 2022-03-17T19:04:58.2312018Z * [new branch] gh/changjishi/2/head -> origin/gh/changjishi/2/head 2022-03-17T19:04:58.2313320Z * [new branch] gh/changjishi/2/orig -> origin/gh/changjishi/2/orig 2022-03-17T19:04:58.2314839Z * [new branch] gh/chauhang/1/base -> origin/gh/chauhang/1/base 2022-03-17T19:04:58.2316299Z * [new branch] gh/chauhang/2/base -> origin/gh/chauhang/2/base 2022-03-17T19:04:58.2318169Z * [new branch] gh/cherie11/1/base -> origin/gh/cherie11/1/base 2022-03-17T19:04:58.2319437Z * [new branch] gh/cherie11/1/head -> origin/gh/cherie11/1/head 2022-03-17T19:04:58.2320799Z * [new branch] gh/cherie11/1/orig -> origin/gh/cherie11/1/orig 2022-03-17T19:04:58.2322948Z * [new branch] gh/chillee/42/base -> origin/gh/chillee/42/base 2022-03-17T19:04:58.2325262Z * [new branch] gh/chillee/42/head -> origin/gh/chillee/42/head 2022-03-17T19:04:58.2326965Z * [new branch] gh/chillee/42/orig -> origin/gh/chillee/42/orig 2022-03-17T19:04:58.2327851Z * [new branch] gh/chillee/47/base -> origin/gh/chillee/47/base 2022-03-17T19:04:58.2329221Z * [new branch] gh/chillee/47/head -> origin/gh/chillee/47/head 2022-03-17T19:04:58.2330424Z * [new branch] gh/chillee/47/orig -> origin/gh/chillee/47/orig 2022-03-17T19:04:58.2332919Z * [new branch] gh/chillee/49/base -> origin/gh/chillee/49/base 2022-03-17T19:04:58.2333864Z * [new branch] gh/chillee/49/head -> origin/gh/chillee/49/head 2022-03-17T19:04:58.2335324Z * [new branch] gh/chillee/49/orig -> origin/gh/chillee/49/orig 2022-03-17T19:04:58.2337289Z * [new branch] gh/choward232/4/base -> origin/gh/choward232/4/base 2022-03-17T19:04:58.2338601Z * [new branch] gh/choward232/4/head -> origin/gh/choward232/4/head 2022-03-17T19:04:58.2340062Z * [new branch] gh/choward232/4/orig -> origin/gh/choward232/4/orig 2022-03-17T19:04:58.2341833Z * [new branch] gh/choward232/6/base -> origin/gh/choward232/6/base 2022-03-17T19:04:58.2343162Z * [new branch] gh/choward232/6/head -> origin/gh/choward232/6/head 2022-03-17T19:04:58.2344606Z * [new branch] gh/choward232/6/orig -> origin/gh/choward232/6/orig 2022-03-17T19:04:58.2346687Z * [new branch] gh/dagitses/100/base -> origin/gh/dagitses/100/base 2022-03-17T19:04:58.2347973Z * [new branch] gh/dagitses/100/head -> origin/gh/dagitses/100/head 2022-03-17T19:04:58.2349570Z * [new branch] gh/dagitses/101/base -> origin/gh/dagitses/101/base 2022-03-17T19:04:58.2350949Z * [new branch] gh/dagitses/101/head -> origin/gh/dagitses/101/head 2022-03-17T19:04:58.2352418Z * [new branch] gh/dagitses/102/base -> origin/gh/dagitses/102/base 2022-03-17T19:04:58.2353865Z * [new branch] gh/dagitses/102/head -> origin/gh/dagitses/102/head 2022-03-17T19:04:58.2355518Z * [new branch] gh/dagitses/103/base -> origin/gh/dagitses/103/base 2022-03-17T19:04:58.2356939Z * [new branch] gh/dagitses/103/head -> origin/gh/dagitses/103/head 2022-03-17T19:04:58.2358473Z * [new branch] gh/dagitses/104/base -> origin/gh/dagitses/104/base 2022-03-17T19:04:58.2359865Z * [new branch] gh/dagitses/104/head -> origin/gh/dagitses/104/head 2022-03-17T19:04:58.2361327Z * [new branch] gh/dagitses/105/base -> origin/gh/dagitses/105/base 2022-03-17T19:04:58.2363093Z * [new branch] gh/dagitses/105/head -> origin/gh/dagitses/105/head 2022-03-17T19:04:58.2364613Z * [new branch] gh/dagitses/106/base -> origin/gh/dagitses/106/base 2022-03-17T19:04:58.2365898Z * [new branch] gh/dagitses/106/head -> origin/gh/dagitses/106/head 2022-03-17T19:04:58.2367856Z * [new branch] gh/dagitses/107/base -> origin/gh/dagitses/107/base 2022-03-17T19:04:58.2369084Z * [new branch] gh/dagitses/107/head -> origin/gh/dagitses/107/head 2022-03-17T19:04:58.2370115Z * [new branch] gh/dagitses/108/base -> origin/gh/dagitses/108/base 2022-03-17T19:04:58.2371159Z * [new branch] gh/dagitses/108/head -> origin/gh/dagitses/108/head 2022-03-17T19:04:58.2373158Z * [new branch] gh/dagitses/109/base -> origin/gh/dagitses/109/base 2022-03-17T19:04:58.2374492Z * [new branch] gh/dagitses/109/head -> origin/gh/dagitses/109/head 2022-03-17T19:04:58.2376272Z * [new branch] gh/dagitses/11/base -> origin/gh/dagitses/11/base 2022-03-17T19:04:58.2377623Z * [new branch] gh/dagitses/11/orig -> origin/gh/dagitses/11/orig 2022-03-17T19:04:58.2379122Z * [new branch] gh/dagitses/110/base -> origin/gh/dagitses/110/base 2022-03-17T19:04:58.2380398Z * [new branch] gh/dagitses/110/head -> origin/gh/dagitses/110/head 2022-03-17T19:04:58.2382203Z * [new branch] gh/dagitses/111/base -> origin/gh/dagitses/111/base 2022-03-17T19:04:58.2383577Z * [new branch] gh/dagitses/111/head -> origin/gh/dagitses/111/head 2022-03-17T19:04:58.2384748Z * [new branch] gh/dagitses/112/base -> origin/gh/dagitses/112/base 2022-03-17T19:04:58.2387055Z * [new branch] gh/dagitses/112/head -> origin/gh/dagitses/112/head 2022-03-17T19:04:58.2387833Z * [new branch] gh/dagitses/113/base -> origin/gh/dagitses/113/base 2022-03-17T19:04:58.2388718Z * [new branch] gh/dagitses/113/head -> origin/gh/dagitses/113/head 2022-03-17T19:04:58.2390214Z * [new branch] gh/dagitses/114/base -> origin/gh/dagitses/114/base 2022-03-17T19:04:58.2391515Z * [new branch] gh/dagitses/114/head -> origin/gh/dagitses/114/head 2022-03-17T19:04:58.2393008Z * [new branch] gh/dagitses/115/base -> origin/gh/dagitses/115/base 2022-03-17T19:04:58.2394278Z * [new branch] gh/dagitses/115/head -> origin/gh/dagitses/115/head 2022-03-17T19:04:58.2395825Z * [new branch] gh/dagitses/116/base -> origin/gh/dagitses/116/base 2022-03-17T19:04:58.2396948Z * [new branch] gh/dagitses/116/head -> origin/gh/dagitses/116/head 2022-03-17T19:04:58.2398131Z * [new branch] gh/dagitses/117/base -> origin/gh/dagitses/117/base 2022-03-17T19:04:58.2400138Z * [new branch] gh/dagitses/117/head -> origin/gh/dagitses/117/head 2022-03-17T19:04:58.2401374Z * [new branch] gh/dagitses/118/base -> origin/gh/dagitses/118/base 2022-03-17T19:04:58.2402776Z * [new branch] gh/dagitses/118/head -> origin/gh/dagitses/118/head 2022-03-17T19:04:58.2404317Z * [new branch] gh/dagitses/119/base -> origin/gh/dagitses/119/base 2022-03-17T19:04:58.2405485Z * [new branch] gh/dagitses/119/head -> origin/gh/dagitses/119/head 2022-03-17T19:04:58.2407400Z * [new branch] gh/dagitses/12/base -> origin/gh/dagitses/12/base 2022-03-17T19:04:58.2408559Z * [new branch] gh/dagitses/12/orig -> origin/gh/dagitses/12/orig 2022-03-17T19:04:58.2409995Z * [new branch] gh/dagitses/120/base -> origin/gh/dagitses/120/base 2022-03-17T19:04:58.2411278Z * [new branch] gh/dagitses/120/head -> origin/gh/dagitses/120/head 2022-03-17T19:04:58.2412760Z * [new branch] gh/dagitses/121/base -> origin/gh/dagitses/121/base 2022-03-17T19:04:58.2414040Z * [new branch] gh/dagitses/121/head -> origin/gh/dagitses/121/head 2022-03-17T19:04:58.2415505Z * [new branch] gh/dagitses/122/base -> origin/gh/dagitses/122/base 2022-03-17T19:04:58.2417465Z * [new branch] gh/dagitses/122/head -> origin/gh/dagitses/122/head 2022-03-17T19:04:58.2418553Z * [new branch] gh/dagitses/123/base -> origin/gh/dagitses/123/base 2022-03-17T19:04:58.2419774Z * [new branch] gh/dagitses/123/head -> origin/gh/dagitses/123/head 2022-03-17T19:04:58.2421296Z * [new branch] gh/dagitses/124/base -> origin/gh/dagitses/124/base 2022-03-17T19:04:58.2422506Z * [new branch] gh/dagitses/124/head -> origin/gh/dagitses/124/head 2022-03-17T19:04:58.2423982Z * [new branch] gh/dagitses/125/base -> origin/gh/dagitses/125/base 2022-03-17T19:04:58.2425227Z * [new branch] gh/dagitses/125/head -> origin/gh/dagitses/125/head 2022-03-17T19:04:58.2426618Z * [new branch] gh/dagitses/125/orig -> origin/gh/dagitses/125/orig 2022-03-17T19:04:58.2428105Z * [new branch] gh/dagitses/126/base -> origin/gh/dagitses/126/base 2022-03-17T19:04:58.2429413Z * [new branch] gh/dagitses/126/head -> origin/gh/dagitses/126/head 2022-03-17T19:04:58.2430881Z * [new branch] gh/dagitses/126/orig -> origin/gh/dagitses/126/orig 2022-03-17T19:04:58.2432392Z * [new branch] gh/dagitses/127/base -> origin/gh/dagitses/127/base 2022-03-17T19:04:58.2433661Z * [new branch] gh/dagitses/127/head -> origin/gh/dagitses/127/head 2022-03-17T19:04:58.2435025Z * [new branch] gh/dagitses/127/orig -> origin/gh/dagitses/127/orig 2022-03-17T19:04:58.2436527Z * [new branch] gh/dagitses/128/base -> origin/gh/dagitses/128/base 2022-03-17T19:04:58.2437769Z * [new branch] gh/dagitses/128/head -> origin/gh/dagitses/128/head 2022-03-17T19:04:58.2439264Z * [new branch] gh/dagitses/129/base -> origin/gh/dagitses/129/base 2022-03-17T19:04:58.2440487Z * [new branch] gh/dagitses/129/head -> origin/gh/dagitses/129/head 2022-03-17T19:04:58.2443072Z * [new branch] gh/dagitses/13/base -> origin/gh/dagitses/13/base 2022-03-17T19:04:58.2443426Z * [new branch] gh/dagitses/13/orig -> origin/gh/dagitses/13/orig 2022-03-17T19:04:58.2444870Z * [new branch] gh/dagitses/130/base -> origin/gh/dagitses/130/base 2022-03-17T19:04:58.2446124Z * [new branch] gh/dagitses/130/head -> origin/gh/dagitses/130/head 2022-03-17T19:04:58.2447594Z * [new branch] gh/dagitses/131/base -> origin/gh/dagitses/131/base 2022-03-17T19:04:58.2448878Z * [new branch] gh/dagitses/131/head -> origin/gh/dagitses/131/head 2022-03-17T19:04:58.2450356Z * [new branch] gh/dagitses/132/base -> origin/gh/dagitses/132/base 2022-03-17T19:04:58.2451658Z * [new branch] gh/dagitses/132/head -> origin/gh/dagitses/132/head 2022-03-17T19:04:58.2452926Z * [new branch] gh/dagitses/132/orig -> origin/gh/dagitses/132/orig 2022-03-17T19:04:58.2454803Z * [new branch] gh/dagitses/133/base -> origin/gh/dagitses/133/base 2022-03-17T19:04:58.2456217Z * [new branch] gh/dagitses/133/head -> origin/gh/dagitses/133/head 2022-03-17T19:04:58.2457960Z * [new branch] gh/dagitses/133/orig -> origin/gh/dagitses/133/orig 2022-03-17T19:04:58.2459625Z * [new branch] gh/dagitses/134/base -> origin/gh/dagitses/134/base 2022-03-17T19:04:58.2461183Z * [new branch] gh/dagitses/134/head -> origin/gh/dagitses/134/head 2022-03-17T19:04:58.2462482Z * [new branch] gh/dagitses/134/orig -> origin/gh/dagitses/134/orig 2022-03-17T19:04:58.2464209Z * [new branch] gh/dagitses/135/base -> origin/gh/dagitses/135/base 2022-03-17T19:04:58.2465457Z * [new branch] gh/dagitses/135/head -> origin/gh/dagitses/135/head 2022-03-17T19:04:58.2466868Z * [new branch] gh/dagitses/135/orig -> origin/gh/dagitses/135/orig 2022-03-17T19:04:58.2468544Z * [new branch] gh/dagitses/136/base -> origin/gh/dagitses/136/base 2022-03-17T19:04:58.2469812Z * [new branch] gh/dagitses/136/head -> origin/gh/dagitses/136/head 2022-03-17T19:04:58.2471042Z * [new branch] gh/dagitses/136/orig -> origin/gh/dagitses/136/orig 2022-03-17T19:04:58.2472941Z * [new branch] gh/dagitses/137/base -> origin/gh/dagitses/137/base 2022-03-17T19:04:58.2474173Z * [new branch] gh/dagitses/137/head -> origin/gh/dagitses/137/head 2022-03-17T19:04:58.2475447Z * [new branch] gh/dagitses/137/orig -> origin/gh/dagitses/137/orig 2022-03-17T19:04:58.2477111Z * [new branch] gh/dagitses/138/base -> origin/gh/dagitses/138/base 2022-03-17T19:04:58.2478533Z * [new branch] gh/dagitses/138/head -> origin/gh/dagitses/138/head 2022-03-17T19:04:58.2479862Z * [new branch] gh/dagitses/138/orig -> origin/gh/dagitses/138/orig 2022-03-17T19:04:58.2481471Z * [new branch] gh/dagitses/139/base -> origin/gh/dagitses/139/base 2022-03-17T19:04:58.2484110Z * [new branch] gh/dagitses/139/head -> origin/gh/dagitses/139/head 2022-03-17T19:04:58.2485295Z * [new branch] gh/dagitses/139/orig -> origin/gh/dagitses/139/orig 2022-03-17T19:04:58.2487040Z * [new branch] gh/dagitses/14/base -> origin/gh/dagitses/14/base 2022-03-17T19:04:58.2488528Z * [new branch] gh/dagitses/14/orig -> origin/gh/dagitses/14/orig 2022-03-17T19:04:58.2490318Z * [new branch] gh/dagitses/140/base -> origin/gh/dagitses/140/base 2022-03-17T19:04:58.2491788Z * [new branch] gh/dagitses/140/head -> origin/gh/dagitses/140/head 2022-03-17T19:04:58.2493050Z * [new branch] gh/dagitses/140/orig -> origin/gh/dagitses/140/orig 2022-03-17T19:04:58.2494717Z * [new branch] gh/dagitses/141/base -> origin/gh/dagitses/141/base 2022-03-17T19:04:58.2496112Z * [new branch] gh/dagitses/141/head -> origin/gh/dagitses/141/head 2022-03-17T19:04:58.2497478Z * [new branch] gh/dagitses/141/orig -> origin/gh/dagitses/141/orig 2022-03-17T19:04:58.2499581Z * [new branch] gh/dagitses/142/base -> origin/gh/dagitses/142/base 2022-03-17T19:04:58.2500677Z * [new branch] gh/dagitses/142/head -> origin/gh/dagitses/142/head 2022-03-17T19:04:58.2502210Z * [new branch] gh/dagitses/142/orig -> origin/gh/dagitses/142/orig 2022-03-17T19:04:58.2503810Z * [new branch] gh/dagitses/143/base -> origin/gh/dagitses/143/base 2022-03-17T19:04:58.2505120Z * [new branch] gh/dagitses/143/head -> origin/gh/dagitses/143/head 2022-03-17T19:04:58.2506410Z * [new branch] gh/dagitses/143/orig -> origin/gh/dagitses/143/orig 2022-03-17T19:04:58.2508044Z * [new branch] gh/dagitses/144/base -> origin/gh/dagitses/144/base 2022-03-17T19:04:58.2509453Z * [new branch] gh/dagitses/144/head -> origin/gh/dagitses/144/head 2022-03-17T19:04:58.2510790Z * [new branch] gh/dagitses/144/orig -> origin/gh/dagitses/144/orig 2022-03-17T19:04:58.2512512Z * [new branch] gh/dagitses/145/base -> origin/gh/dagitses/145/base 2022-03-17T19:04:58.2513893Z * [new branch] gh/dagitses/145/head -> origin/gh/dagitses/145/head 2022-03-17T19:04:58.2514951Z * [new branch] gh/dagitses/145/orig -> origin/gh/dagitses/145/orig 2022-03-17T19:04:58.2516692Z * [new branch] gh/dagitses/146/base -> origin/gh/dagitses/146/base 2022-03-17T19:04:58.2518020Z * [new branch] gh/dagitses/146/head -> origin/gh/dagitses/146/head 2022-03-17T19:04:58.2519378Z * [new branch] gh/dagitses/146/orig -> origin/gh/dagitses/146/orig 2022-03-17T19:04:58.2521071Z * [new branch] gh/dagitses/147/base -> origin/gh/dagitses/147/base 2022-03-17T19:04:58.2522238Z * [new branch] gh/dagitses/147/head -> origin/gh/dagitses/147/head 2022-03-17T19:04:58.2523859Z * [new branch] gh/dagitses/148/base -> origin/gh/dagitses/148/base 2022-03-17T19:04:58.2525145Z * [new branch] gh/dagitses/148/head -> origin/gh/dagitses/148/head 2022-03-17T19:04:58.2526653Z * [new branch] gh/dagitses/149/base -> origin/gh/dagitses/149/base 2022-03-17T19:04:58.2528026Z * [new branch] gh/dagitses/149/head -> origin/gh/dagitses/149/head 2022-03-17T19:04:58.2529777Z * [new branch] gh/dagitses/15/base -> origin/gh/dagitses/15/base 2022-03-17T19:04:58.2531059Z * [new branch] gh/dagitses/15/orig -> origin/gh/dagitses/15/orig 2022-03-17T19:04:58.2532568Z * [new branch] gh/dagitses/150/base -> origin/gh/dagitses/150/base 2022-03-17T19:04:58.2533898Z * [new branch] gh/dagitses/150/head -> origin/gh/dagitses/150/head 2022-03-17T19:04:58.2535467Z * [new branch] gh/dagitses/151/base -> origin/gh/dagitses/151/base 2022-03-17T19:04:58.2536787Z * [new branch] gh/dagitses/151/head -> origin/gh/dagitses/151/head 2022-03-17T19:04:58.2538249Z * [new branch] gh/dagitses/152/base -> origin/gh/dagitses/152/base 2022-03-17T19:04:58.2539550Z * [new branch] gh/dagitses/152/head -> origin/gh/dagitses/152/head 2022-03-17T19:04:58.2541039Z * [new branch] gh/dagitses/153/base -> origin/gh/dagitses/153/base 2022-03-17T19:04:58.2542393Z * [new branch] gh/dagitses/153/head -> origin/gh/dagitses/153/head 2022-03-17T19:04:58.2543931Z * [new branch] gh/dagitses/154/base -> origin/gh/dagitses/154/base 2022-03-17T19:04:58.2545365Z * [new branch] gh/dagitses/154/head -> origin/gh/dagitses/154/head 2022-03-17T19:04:58.2546848Z * [new branch] gh/dagitses/155/base -> origin/gh/dagitses/155/base 2022-03-17T19:04:58.2548188Z * [new branch] gh/dagitses/155/head -> origin/gh/dagitses/155/head 2022-03-17T19:04:58.2549691Z * [new branch] gh/dagitses/156/base -> origin/gh/dagitses/156/base 2022-03-17T19:04:58.2551181Z * [new branch] gh/dagitses/156/head -> origin/gh/dagitses/156/head 2022-03-17T19:04:58.2552592Z * [new branch] gh/dagitses/157/base -> origin/gh/dagitses/157/base 2022-03-17T19:04:58.2553951Z * [new branch] gh/dagitses/157/head -> origin/gh/dagitses/157/head 2022-03-17T19:04:58.2555483Z * [new branch] gh/dagitses/158/base -> origin/gh/dagitses/158/base 2022-03-17T19:04:58.2556902Z * [new branch] gh/dagitses/158/head -> origin/gh/dagitses/158/head 2022-03-17T19:04:58.2558368Z * [new branch] gh/dagitses/159/base -> origin/gh/dagitses/159/base 2022-03-17T19:04:58.2559652Z * [new branch] gh/dagitses/159/head -> origin/gh/dagitses/159/head 2022-03-17T19:04:58.2561360Z * [new branch] gh/dagitses/16/base -> origin/gh/dagitses/16/base 2022-03-17T19:04:58.2563046Z * [new branch] gh/dagitses/16/orig -> origin/gh/dagitses/16/orig 2022-03-17T19:04:58.2564589Z * [new branch] gh/dagitses/160/base -> origin/gh/dagitses/160/base 2022-03-17T19:04:58.2565904Z * [new branch] gh/dagitses/160/head -> origin/gh/dagitses/160/head 2022-03-17T19:04:58.2567430Z * [new branch] gh/dagitses/161/base -> origin/gh/dagitses/161/base 2022-03-17T19:04:58.2568771Z * [new branch] gh/dagitses/161/head -> origin/gh/dagitses/161/head 2022-03-17T19:04:58.2570248Z * [new branch] gh/dagitses/162/base -> origin/gh/dagitses/162/base 2022-03-17T19:04:58.2571403Z * [new branch] gh/dagitses/162/head -> origin/gh/dagitses/162/head 2022-03-17T19:04:58.2573161Z * [new branch] gh/dagitses/163/base -> origin/gh/dagitses/163/base 2022-03-17T19:04:58.2574230Z * [new branch] gh/dagitses/163/head -> origin/gh/dagitses/163/head 2022-03-17T19:04:58.2575840Z * [new branch] gh/dagitses/164/base -> origin/gh/dagitses/164/base 2022-03-17T19:04:58.2577208Z * [new branch] gh/dagitses/164/head -> origin/gh/dagitses/164/head 2022-03-17T19:04:58.2578627Z * [new branch] gh/dagitses/165/base -> origin/gh/dagitses/165/base 2022-03-17T19:04:58.2579948Z * [new branch] gh/dagitses/165/head -> origin/gh/dagitses/165/head 2022-03-17T19:04:58.2581409Z * [new branch] gh/dagitses/166/base -> origin/gh/dagitses/166/base 2022-03-17T19:04:58.2582562Z * [new branch] gh/dagitses/166/head -> origin/gh/dagitses/166/head 2022-03-17T19:04:58.2584094Z * [new branch] gh/dagitses/167/base -> origin/gh/dagitses/167/base 2022-03-17T19:04:58.2585451Z * [new branch] gh/dagitses/167/head -> origin/gh/dagitses/167/head 2022-03-17T19:04:58.2587079Z * [new branch] gh/dagitses/168/base -> origin/gh/dagitses/168/base 2022-03-17T19:04:58.2588368Z * [new branch] gh/dagitses/168/head -> origin/gh/dagitses/168/head 2022-03-17T19:04:58.2589894Z * [new branch] gh/dagitses/169/base -> origin/gh/dagitses/169/base 2022-03-17T19:04:58.2591180Z * [new branch] gh/dagitses/169/head -> origin/gh/dagitses/169/head 2022-03-17T19:04:58.2592901Z * [new branch] gh/dagitses/17/base -> origin/gh/dagitses/17/base 2022-03-17T19:04:58.2594280Z * [new branch] gh/dagitses/17/orig -> origin/gh/dagitses/17/orig 2022-03-17T19:04:58.2595711Z * [new branch] gh/dagitses/170/base -> origin/gh/dagitses/170/base 2022-03-17T19:04:58.2596992Z * [new branch] gh/dagitses/170/head -> origin/gh/dagitses/170/head 2022-03-17T19:04:58.2598594Z * [new branch] gh/dagitses/171/base -> origin/gh/dagitses/171/base 2022-03-17T19:04:58.2599885Z * [new branch] gh/dagitses/171/head -> origin/gh/dagitses/171/head 2022-03-17T19:04:58.2601350Z * [new branch] gh/dagitses/172/base -> origin/gh/dagitses/172/base 2022-03-17T19:04:58.2603980Z * [new branch] gh/dagitses/172/head -> origin/gh/dagitses/172/head 2022-03-17T19:04:58.2604518Z * [new branch] gh/dagitses/173/base -> origin/gh/dagitses/173/base 2022-03-17T19:04:58.2605463Z * [new branch] gh/dagitses/173/head -> origin/gh/dagitses/173/head 2022-03-17T19:04:58.2607073Z * [new branch] gh/dagitses/174/base -> origin/gh/dagitses/174/base 2022-03-17T19:04:58.2608365Z * [new branch] gh/dagitses/174/head -> origin/gh/dagitses/174/head 2022-03-17T19:04:58.2609924Z * [new branch] gh/dagitses/175/base -> origin/gh/dagitses/175/base 2022-03-17T19:04:58.2611038Z * [new branch] gh/dagitses/175/head -> origin/gh/dagitses/175/head 2022-03-17T19:04:58.2612623Z * [new branch] gh/dagitses/176/base -> origin/gh/dagitses/176/base 2022-03-17T19:04:58.2613801Z * [new branch] gh/dagitses/176/head -> origin/gh/dagitses/176/head 2022-03-17T19:04:58.2615418Z * [new branch] gh/dagitses/177/base -> origin/gh/dagitses/177/base 2022-03-17T19:04:58.2616693Z * [new branch] gh/dagitses/177/head -> origin/gh/dagitses/177/head 2022-03-17T19:04:58.2618213Z * [new branch] gh/dagitses/178/base -> origin/gh/dagitses/178/base 2022-03-17T19:04:58.2619572Z * [new branch] gh/dagitses/178/head -> origin/gh/dagitses/178/head 2022-03-17T19:04:58.2621185Z * [new branch] gh/dagitses/179/base -> origin/gh/dagitses/179/base 2022-03-17T19:04:58.2622280Z * [new branch] gh/dagitses/179/head -> origin/gh/dagitses/179/head 2022-03-17T19:04:58.2624086Z * [new branch] gh/dagitses/18/base -> origin/gh/dagitses/18/base 2022-03-17T19:04:58.2625247Z * [new branch] gh/dagitses/18/orig -> origin/gh/dagitses/18/orig 2022-03-17T19:04:58.2626810Z * [new branch] gh/dagitses/180/base -> origin/gh/dagitses/180/base 2022-03-17T19:04:58.2628079Z * [new branch] gh/dagitses/180/head -> origin/gh/dagitses/180/head 2022-03-17T19:04:58.2629584Z * [new branch] gh/dagitses/181/base -> origin/gh/dagitses/181/base 2022-03-17T19:04:58.2630967Z * [new branch] gh/dagitses/181/head -> origin/gh/dagitses/181/head 2022-03-17T19:04:58.2632459Z * [new branch] gh/dagitses/182/base -> origin/gh/dagitses/182/base 2022-03-17T19:04:58.2634035Z * [new branch] gh/dagitses/182/head -> origin/gh/dagitses/182/head 2022-03-17T19:04:58.2635597Z * [new branch] gh/dagitses/183/base -> origin/gh/dagitses/183/base 2022-03-17T19:04:58.2636808Z * [new branch] gh/dagitses/183/head -> origin/gh/dagitses/183/head 2022-03-17T19:04:58.2638402Z * [new branch] gh/dagitses/184/base -> origin/gh/dagitses/184/base 2022-03-17T19:04:58.2639703Z * [new branch] gh/dagitses/184/head -> origin/gh/dagitses/184/head 2022-03-17T19:04:58.2641234Z * [new branch] gh/dagitses/185/base -> origin/gh/dagitses/185/base 2022-03-17T19:04:58.2642662Z * [new branch] gh/dagitses/185/head -> origin/gh/dagitses/185/head 2022-03-17T19:04:58.2648008Z * [new branch] gh/dagitses/186/base -> origin/gh/dagitses/186/base 2022-03-17T19:04:58.2651775Z * [new branch] gh/dagitses/186/head -> origin/gh/dagitses/186/head 2022-03-17T19:04:58.2656453Z * [new branch] gh/dagitses/187/base -> origin/gh/dagitses/187/base 2022-03-17T19:04:58.2659546Z * [new branch] gh/dagitses/187/head -> origin/gh/dagitses/187/head 2022-03-17T19:04:58.2663626Z * [new branch] gh/dagitses/188/base -> origin/gh/dagitses/188/base 2022-03-17T19:04:58.2666965Z * [new branch] gh/dagitses/188/head -> origin/gh/dagitses/188/head 2022-03-17T19:04:58.2670972Z * [new branch] gh/dagitses/189/base -> origin/gh/dagitses/189/base 2022-03-17T19:04:58.2674312Z * [new branch] gh/dagitses/189/head -> origin/gh/dagitses/189/head 2022-03-17T19:04:58.2678837Z * [new branch] gh/dagitses/19/base -> origin/gh/dagitses/19/base 2022-03-17T19:04:58.2682469Z * [new branch] gh/dagitses/19/orig -> origin/gh/dagitses/19/orig 2022-03-17T19:04:58.2687140Z * [new branch] gh/dagitses/190/base -> origin/gh/dagitses/190/base 2022-03-17T19:04:58.2690371Z * [new branch] gh/dagitses/190/head -> origin/gh/dagitses/190/head 2022-03-17T19:04:58.2694552Z * [new branch] gh/dagitses/191/base -> origin/gh/dagitses/191/base 2022-03-17T19:04:58.2697889Z * [new branch] gh/dagitses/191/head -> origin/gh/dagitses/191/head 2022-03-17T19:04:58.2701868Z * [new branch] gh/dagitses/192/base -> origin/gh/dagitses/192/base 2022-03-17T19:04:58.2705246Z * [new branch] gh/dagitses/192/head -> origin/gh/dagitses/192/head 2022-03-17T19:04:58.2709168Z * [new branch] gh/dagitses/193/base -> origin/gh/dagitses/193/base 2022-03-17T19:04:58.2712580Z * [new branch] gh/dagitses/193/head -> origin/gh/dagitses/193/head 2022-03-17T19:04:58.2716532Z * [new branch] gh/dagitses/194/base -> origin/gh/dagitses/194/base 2022-03-17T19:04:58.2719875Z * [new branch] gh/dagitses/194/head -> origin/gh/dagitses/194/head 2022-03-17T19:04:58.2724412Z * [new branch] gh/dagitses/195/base -> origin/gh/dagitses/195/base 2022-03-17T19:04:58.2728130Z * [new branch] gh/dagitses/195/head -> origin/gh/dagitses/195/head 2022-03-17T19:04:58.2732042Z * [new branch] gh/dagitses/196/base -> origin/gh/dagitses/196/base 2022-03-17T19:04:58.2736319Z * [new branch] gh/dagitses/196/head -> origin/gh/dagitses/196/head 2022-03-17T19:04:58.2740649Z * [new branch] gh/dagitses/197/base -> origin/gh/dagitses/197/base 2022-03-17T19:04:58.2744420Z * [new branch] gh/dagitses/197/head -> origin/gh/dagitses/197/head 2022-03-17T19:04:58.2748878Z * [new branch] gh/dagitses/198/base -> origin/gh/dagitses/198/base 2022-03-17T19:04:58.2752288Z * [new branch] gh/dagitses/198/head -> origin/gh/dagitses/198/head 2022-03-17T19:04:58.2756263Z * [new branch] gh/dagitses/199/base -> origin/gh/dagitses/199/base 2022-03-17T19:04:58.2759690Z * [new branch] gh/dagitses/199/head -> origin/gh/dagitses/199/head 2022-03-17T19:04:58.2765009Z * [new branch] gh/dagitses/20/base -> origin/gh/dagitses/20/base 2022-03-17T19:04:58.2768642Z * [new branch] gh/dagitses/20/orig -> origin/gh/dagitses/20/orig 2022-03-17T19:04:58.2770753Z * [new branch] gh/dagitses/200/base -> origin/gh/dagitses/200/base 2022-03-17T19:04:58.2771441Z * [new branch] gh/dagitses/200/head -> origin/gh/dagitses/200/head 2022-03-17T19:04:58.2773155Z * [new branch] gh/dagitses/201/base -> origin/gh/dagitses/201/base 2022-03-17T19:04:58.2774371Z * [new branch] gh/dagitses/201/head -> origin/gh/dagitses/201/head 2022-03-17T19:04:58.2776291Z * [new branch] gh/dagitses/202/base -> origin/gh/dagitses/202/base 2022-03-17T19:04:58.2777467Z * [new branch] gh/dagitses/202/head -> origin/gh/dagitses/202/head 2022-03-17T19:04:58.2779196Z * [new branch] gh/dagitses/203/base -> origin/gh/dagitses/203/base 2022-03-17T19:04:58.2780593Z * [new branch] gh/dagitses/203/head -> origin/gh/dagitses/203/head 2022-03-17T19:04:58.2781997Z * [new branch] gh/dagitses/204/base -> origin/gh/dagitses/204/base 2022-03-17T19:04:58.2783458Z * [new branch] gh/dagitses/204/head -> origin/gh/dagitses/204/head 2022-03-17T19:04:58.2785030Z * [new branch] gh/dagitses/205/base -> origin/gh/dagitses/205/base 2022-03-17T19:04:58.2786504Z * [new branch] gh/dagitses/205/head -> origin/gh/dagitses/205/head 2022-03-17T19:04:58.2788082Z * [new branch] gh/dagitses/206/base -> origin/gh/dagitses/206/base 2022-03-17T19:04:58.2789264Z * [new branch] gh/dagitses/206/head -> origin/gh/dagitses/206/head 2022-03-17T19:04:58.2790993Z * [new branch] gh/dagitses/207/base -> origin/gh/dagitses/207/base 2022-03-17T19:04:58.2792192Z * [new branch] gh/dagitses/207/head -> origin/gh/dagitses/207/head 2022-03-17T19:04:58.2793858Z * [new branch] gh/dagitses/208/base -> origin/gh/dagitses/208/base 2022-03-17T19:04:58.2795257Z * [new branch] gh/dagitses/208/head -> origin/gh/dagitses/208/head 2022-03-17T19:04:58.2796781Z * [new branch] gh/dagitses/209/base -> origin/gh/dagitses/209/base 2022-03-17T19:04:58.2798203Z * [new branch] gh/dagitses/209/head -> origin/gh/dagitses/209/head 2022-03-17T19:04:58.2799879Z * [new branch] gh/dagitses/21/base -> origin/gh/dagitses/21/base 2022-03-17T19:04:58.2801282Z * [new branch] gh/dagitses/21/orig -> origin/gh/dagitses/21/orig 2022-03-17T19:04:58.2803006Z * [new branch] gh/dagitses/210/base -> origin/gh/dagitses/210/base 2022-03-17T19:04:58.2804153Z * [new branch] gh/dagitses/210/head -> origin/gh/dagitses/210/head 2022-03-17T19:04:58.2806249Z * [new branch] gh/dagitses/211/base -> origin/gh/dagitses/211/base 2022-03-17T19:04:58.2808127Z * [new branch] gh/dagitses/211/head -> origin/gh/dagitses/211/head 2022-03-17T19:04:58.2810013Z * [new branch] gh/dagitses/212/base -> origin/gh/dagitses/212/base 2022-03-17T19:04:58.2811418Z * [new branch] gh/dagitses/212/head -> origin/gh/dagitses/212/head 2022-03-17T19:04:58.2813218Z * [new branch] gh/dagitses/213/base -> origin/gh/dagitses/213/base 2022-03-17T19:04:58.2814899Z * [new branch] gh/dagitses/213/head -> origin/gh/dagitses/213/head 2022-03-17T19:04:58.2816807Z * [new branch] gh/dagitses/214/base -> origin/gh/dagitses/214/base 2022-03-17T19:04:58.2818210Z * [new branch] gh/dagitses/214/head -> origin/gh/dagitses/214/head 2022-03-17T19:04:58.2820071Z * [new branch] gh/dagitses/215/base -> origin/gh/dagitses/215/base 2022-03-17T19:04:58.2821103Z * [new branch] gh/dagitses/215/head -> origin/gh/dagitses/215/head 2022-03-17T19:04:58.2822852Z * [new branch] gh/dagitses/216/base -> origin/gh/dagitses/216/base 2022-03-17T19:04:58.2823933Z * [new branch] gh/dagitses/216/head -> origin/gh/dagitses/216/head 2022-03-17T19:04:58.2825608Z * [new branch] gh/dagitses/217/base -> origin/gh/dagitses/217/base 2022-03-17T19:04:58.2826722Z * [new branch] gh/dagitses/217/head -> origin/gh/dagitses/217/head 2022-03-17T19:04:58.2828432Z * [new branch] gh/dagitses/218/base -> origin/gh/dagitses/218/base 2022-03-17T19:04:58.2829485Z * [new branch] gh/dagitses/218/head -> origin/gh/dagitses/218/head 2022-03-17T19:04:58.2831145Z * [new branch] gh/dagitses/219/base -> origin/gh/dagitses/219/base 2022-03-17T19:04:58.2832238Z * [new branch] gh/dagitses/219/head -> origin/gh/dagitses/219/head 2022-03-17T19:04:58.2833936Z * [new branch] gh/dagitses/220/base -> origin/gh/dagitses/220/base 2022-03-17T19:04:58.2835108Z * [new branch] gh/dagitses/220/head -> origin/gh/dagitses/220/head 2022-03-17T19:04:58.2836725Z * [new branch] gh/dagitses/221/base -> origin/gh/dagitses/221/base 2022-03-17T19:04:58.2837888Z * [new branch] gh/dagitses/221/head -> origin/gh/dagitses/221/head 2022-03-17T19:04:58.2839470Z * [new branch] gh/dagitses/222/base -> origin/gh/dagitses/222/base 2022-03-17T19:04:58.2840776Z * [new branch] gh/dagitses/222/head -> origin/gh/dagitses/222/head 2022-03-17T19:04:58.2842372Z * [new branch] gh/dagitses/223/base -> origin/gh/dagitses/223/base 2022-03-17T19:04:58.2844166Z * [new branch] gh/dagitses/223/head -> origin/gh/dagitses/223/head 2022-03-17T19:04:58.2845627Z * [new branch] gh/dagitses/224/base -> origin/gh/dagitses/224/base 2022-03-17T19:04:58.2846921Z * [new branch] gh/dagitses/224/head -> origin/gh/dagitses/224/head 2022-03-17T19:04:58.2848253Z * [new branch] gh/dagitses/224/orig -> origin/gh/dagitses/224/orig 2022-03-17T19:04:58.2849847Z * [new branch] gh/dagitses/225/base -> origin/gh/dagitses/225/base 2022-03-17T19:04:58.2851149Z * [new branch] gh/dagitses/225/head -> origin/gh/dagitses/225/head 2022-03-17T19:04:58.2852497Z * [new branch] gh/dagitses/225/orig -> origin/gh/dagitses/225/orig 2022-03-17T19:04:58.2854038Z * [new branch] gh/dagitses/226/base -> origin/gh/dagitses/226/base 2022-03-17T19:04:58.2855380Z * [new branch] gh/dagitses/226/head -> origin/gh/dagitses/226/head 2022-03-17T19:04:58.2856946Z * [new branch] gh/dagitses/226/orig -> origin/gh/dagitses/226/orig 2022-03-17T19:04:58.2858550Z * [new branch] gh/dagitses/227/base -> origin/gh/dagitses/227/base 2022-03-17T19:04:58.2859716Z * [new branch] gh/dagitses/227/head -> origin/gh/dagitses/227/head 2022-03-17T19:04:58.2861391Z * [new branch] gh/dagitses/227/orig -> origin/gh/dagitses/227/orig 2022-03-17T19:04:58.2862869Z * [new branch] gh/dagitses/228/base -> origin/gh/dagitses/228/base 2022-03-17T19:04:58.2864007Z * [new branch] gh/dagitses/228/head -> origin/gh/dagitses/228/head 2022-03-17T19:04:58.2865290Z * [new branch] gh/dagitses/228/orig -> origin/gh/dagitses/228/orig 2022-03-17T19:04:58.2866836Z * [new branch] gh/dagitses/229/base -> origin/gh/dagitses/229/base 2022-03-17T19:04:58.2868201Z * [new branch] gh/dagitses/229/head -> origin/gh/dagitses/229/head 2022-03-17T19:04:58.2869613Z * [new branch] gh/dagitses/229/orig -> origin/gh/dagitses/229/orig 2022-03-17T19:04:58.2871323Z * [new branch] gh/dagitses/23/base -> origin/gh/dagitses/23/base 2022-03-17T19:04:58.2872716Z * [new branch] gh/dagitses/23/orig -> origin/gh/dagitses/23/orig 2022-03-17T19:04:58.2874323Z * [new branch] gh/dagitses/230/base -> origin/gh/dagitses/230/base 2022-03-17T19:04:58.2875598Z * [new branch] gh/dagitses/230/head -> origin/gh/dagitses/230/head 2022-03-17T19:04:58.2876854Z * [new branch] gh/dagitses/230/orig -> origin/gh/dagitses/230/orig 2022-03-17T19:04:58.2878398Z * [new branch] gh/dagitses/231/base -> origin/gh/dagitses/231/base 2022-03-17T19:04:58.2879825Z * [new branch] gh/dagitses/231/head -> origin/gh/dagitses/231/head 2022-03-17T19:04:58.2881281Z * [new branch] gh/dagitses/231/orig -> origin/gh/dagitses/231/orig 2022-03-17T19:04:58.2882973Z * [new branch] gh/dagitses/232/base -> origin/gh/dagitses/232/base 2022-03-17T19:04:58.2884341Z * [new branch] gh/dagitses/232/head -> origin/gh/dagitses/232/head 2022-03-17T19:04:58.2885731Z * [new branch] gh/dagitses/232/orig -> origin/gh/dagitses/232/orig 2022-03-17T19:04:58.2887327Z * [new branch] gh/dagitses/233/base -> origin/gh/dagitses/233/base 2022-03-17T19:04:58.2888414Z * [new branch] gh/dagitses/233/head -> origin/gh/dagitses/233/head 2022-03-17T19:04:58.2889700Z * [new branch] gh/dagitses/233/orig -> origin/gh/dagitses/233/orig 2022-03-17T19:04:58.2891241Z * [new branch] gh/dagitses/234/base -> origin/gh/dagitses/234/base 2022-03-17T19:04:58.2892563Z * [new branch] gh/dagitses/234/head -> origin/gh/dagitses/234/head 2022-03-17T19:04:58.2893895Z * [new branch] gh/dagitses/234/orig -> origin/gh/dagitses/234/orig 2022-03-17T19:04:58.2895466Z * [new branch] gh/dagitses/235/base -> origin/gh/dagitses/235/base 2022-03-17T19:04:58.2896806Z * [new branch] gh/dagitses/235/head -> origin/gh/dagitses/235/head 2022-03-17T19:04:58.2898184Z * [new branch] gh/dagitses/235/orig -> origin/gh/dagitses/235/orig 2022-03-17T19:04:58.2899775Z * [new branch] gh/dagitses/236/base -> origin/gh/dagitses/236/base 2022-03-17T19:04:58.2901110Z * [new branch] gh/dagitses/236/head -> origin/gh/dagitses/236/head 2022-03-17T19:04:58.2902420Z * [new branch] gh/dagitses/236/orig -> origin/gh/dagitses/236/orig 2022-03-17T19:04:58.2903988Z * [new branch] gh/dagitses/237/base -> origin/gh/dagitses/237/base 2022-03-17T19:04:58.2905325Z * [new branch] gh/dagitses/237/head -> origin/gh/dagitses/237/head 2022-03-17T19:04:58.2906674Z * [new branch] gh/dagitses/237/orig -> origin/gh/dagitses/237/orig 2022-03-17T19:04:58.2908278Z * [new branch] gh/dagitses/238/base -> origin/gh/dagitses/238/base 2022-03-17T19:04:58.2909613Z * [new branch] gh/dagitses/238/head -> origin/gh/dagitses/238/head 2022-03-17T19:04:58.2911113Z * [new branch] gh/dagitses/238/orig -> origin/gh/dagitses/238/orig 2022-03-17T19:04:58.2912921Z * [new branch] gh/dagitses/239/base -> origin/gh/dagitses/239/base 2022-03-17T19:04:58.2914278Z * [new branch] gh/dagitses/239/head -> origin/gh/dagitses/239/head 2022-03-17T19:04:58.2915490Z * [new branch] gh/dagitses/239/orig -> origin/gh/dagitses/239/orig 2022-03-17T19:04:58.2917338Z * [new branch] gh/dagitses/240/base -> origin/gh/dagitses/240/base 2022-03-17T19:04:58.2918606Z * [new branch] gh/dagitses/240/head -> origin/gh/dagitses/240/head 2022-03-17T19:04:58.2919925Z * [new branch] gh/dagitses/240/orig -> origin/gh/dagitses/240/orig 2022-03-17T19:04:58.2921755Z * [new branch] gh/dagitses/241/base -> origin/gh/dagitses/241/base 2022-03-17T19:04:58.2923464Z * [new branch] gh/dagitses/241/head -> origin/gh/dagitses/241/head 2022-03-17T19:04:58.2924714Z * [new branch] gh/dagitses/241/orig -> origin/gh/dagitses/241/orig 2022-03-17T19:04:58.2926598Z * [new branch] gh/dagitses/244/base -> origin/gh/dagitses/244/base 2022-03-17T19:04:58.2927943Z * [new branch] gh/dagitses/244/head -> origin/gh/dagitses/244/head 2022-03-17T19:04:58.2929247Z * [new branch] gh/dagitses/244/orig -> origin/gh/dagitses/244/orig 2022-03-17T19:04:58.2930987Z * [new branch] gh/dagitses/245/base -> origin/gh/dagitses/245/base 2022-03-17T19:04:58.2932460Z * [new branch] gh/dagitses/245/head -> origin/gh/dagitses/245/head 2022-03-17T19:04:58.2933955Z * [new branch] gh/dagitses/245/orig -> origin/gh/dagitses/245/orig 2022-03-17T19:04:58.2935964Z * [new branch] gh/dagitses/25/base -> origin/gh/dagitses/25/base 2022-03-17T19:04:58.2937490Z * [new branch] gh/dagitses/25/orig -> origin/gh/dagitses/25/orig 2022-03-17T19:04:58.2939569Z * [new branch] gh/dagitses/257/base -> origin/gh/dagitses/257/base 2022-03-17T19:04:58.2940953Z * [new branch] gh/dagitses/257/head -> origin/gh/dagitses/257/head 2022-03-17T19:04:58.2942472Z * [new branch] gh/dagitses/257/orig -> origin/gh/dagitses/257/orig 2022-03-17T19:04:58.2944507Z * [new branch] gh/dagitses/26/base -> origin/gh/dagitses/26/base 2022-03-17T19:04:58.2945986Z * [new branch] gh/dagitses/26/orig -> origin/gh/dagitses/26/orig 2022-03-17T19:04:58.2948230Z * [new branch] gh/dagitses/262/base -> origin/gh/dagitses/262/base 2022-03-17T19:04:58.2949674Z * [new branch] gh/dagitses/262/head -> origin/gh/dagitses/262/head 2022-03-17T19:04:58.2951126Z * [new branch] gh/dagitses/262/orig -> origin/gh/dagitses/262/orig 2022-03-17T19:04:58.2953602Z * [new branch] gh/dagitses/263/base -> origin/gh/dagitses/263/base 2022-03-17T19:04:58.2954616Z * [new branch] gh/dagitses/263/head -> origin/gh/dagitses/263/head 2022-03-17T19:04:58.2956091Z * [new branch] gh/dagitses/263/orig -> origin/gh/dagitses/263/orig 2022-03-17T19:04:58.2957786Z * [new branch] gh/dagitses/264/base -> origin/gh/dagitses/264/base 2022-03-17T19:04:58.2959192Z * [new branch] gh/dagitses/264/head -> origin/gh/dagitses/264/head 2022-03-17T19:04:58.2960593Z * [new branch] gh/dagitses/264/orig -> origin/gh/dagitses/264/orig 2022-03-17T19:04:58.2962658Z * [new branch] gh/dagitses/265/base -> origin/gh/dagitses/265/base 2022-03-17T19:04:58.2964244Z * [new branch] gh/dagitses/265/head -> origin/gh/dagitses/265/head 2022-03-17T19:04:58.2965737Z * [new branch] gh/dagitses/265/orig -> origin/gh/dagitses/265/orig 2022-03-17T19:04:58.2967707Z * [new branch] gh/dagitses/266/base -> origin/gh/dagitses/266/base 2022-03-17T19:04:58.2969370Z * [new branch] gh/dagitses/266/head -> origin/gh/dagitses/266/head 2022-03-17T19:04:58.2970564Z * [new branch] gh/dagitses/266/orig -> origin/gh/dagitses/266/orig 2022-03-17T19:04:58.2972390Z * [new branch] gh/dagitses/271/base -> origin/gh/dagitses/271/base 2022-03-17T19:04:58.2973571Z * [new branch] gh/dagitses/271/head -> origin/gh/dagitses/271/head 2022-03-17T19:04:58.2975210Z * [new branch] gh/dagitses/271/orig -> origin/gh/dagitses/271/orig 2022-03-17T19:04:58.2976946Z * [new branch] gh/dagitses/272/base -> origin/gh/dagitses/272/base 2022-03-17T19:04:58.2978308Z * [new branch] gh/dagitses/272/head -> origin/gh/dagitses/272/head 2022-03-17T19:04:58.2979665Z * [new branch] gh/dagitses/272/orig -> origin/gh/dagitses/272/orig 2022-03-17T19:04:58.2981461Z * [new branch] gh/dagitses/273/base -> origin/gh/dagitses/273/base 2022-03-17T19:04:58.2982911Z * [new branch] gh/dagitses/273/head -> origin/gh/dagitses/273/head 2022-03-17T19:04:58.2984144Z * [new branch] gh/dagitses/273/orig -> origin/gh/dagitses/273/orig 2022-03-17T19:04:58.2986194Z * [new branch] gh/dagitses/275/base -> origin/gh/dagitses/275/base 2022-03-17T19:04:58.2987664Z * [new branch] gh/dagitses/275/head -> origin/gh/dagitses/275/head 2022-03-17T19:04:58.2988765Z * [new branch] gh/dagitses/275/orig -> origin/gh/dagitses/275/orig 2022-03-17T19:04:58.2990464Z * [new branch] gh/dagitses/277/base -> origin/gh/dagitses/277/base 2022-03-17T19:04:58.2991888Z * [new branch] gh/dagitses/277/head -> origin/gh/dagitses/277/head 2022-03-17T19:04:58.2993235Z * [new branch] gh/dagitses/277/orig -> origin/gh/dagitses/277/orig 2022-03-17T19:04:58.2994974Z * [new branch] gh/dagitses/278/base -> origin/gh/dagitses/278/base 2022-03-17T19:04:58.2996376Z * [new branch] gh/dagitses/278/head -> origin/gh/dagitses/278/head 2022-03-17T19:04:58.2997880Z * [new branch] gh/dagitses/278/orig -> origin/gh/dagitses/278/orig 2022-03-17T19:04:58.2999451Z * [new branch] gh/dagitses/279/base -> origin/gh/dagitses/279/base 2022-03-17T19:04:58.3000836Z * [new branch] gh/dagitses/279/orig -> origin/gh/dagitses/279/orig 2022-03-17T19:04:58.3002761Z * [new branch] gh/dagitses/280/base -> origin/gh/dagitses/280/base 2022-03-17T19:04:58.3004234Z * [new branch] gh/dagitses/280/orig -> origin/gh/dagitses/280/orig 2022-03-17T19:04:58.3005786Z * [new branch] gh/dagitses/281/base -> origin/gh/dagitses/281/base 2022-03-17T19:04:58.3007120Z * [new branch] gh/dagitses/281/orig -> origin/gh/dagitses/281/orig 2022-03-17T19:04:58.3008869Z * [new branch] gh/dagitses/282/base -> origin/gh/dagitses/282/base 2022-03-17T19:04:58.3010118Z * [new branch] gh/dagitses/282/orig -> origin/gh/dagitses/282/orig 2022-03-17T19:04:58.3011942Z * [new branch] gh/dagitses/283/base -> origin/gh/dagitses/283/base 2022-03-17T19:04:58.3013432Z * [new branch] gh/dagitses/283/orig -> origin/gh/dagitses/283/orig 2022-03-17T19:04:58.3015237Z * [new branch] gh/dagitses/284/base -> origin/gh/dagitses/284/base 2022-03-17T19:04:58.3016621Z * [new branch] gh/dagitses/284/orig -> origin/gh/dagitses/284/orig 2022-03-17T19:04:58.3018363Z * [new branch] gh/dagitses/285/base -> origin/gh/dagitses/285/base 2022-03-17T19:04:58.3019695Z * [new branch] gh/dagitses/285/orig -> origin/gh/dagitses/285/orig 2022-03-17T19:04:58.3021375Z * [new branch] gh/dagitses/286/base -> origin/gh/dagitses/286/base 2022-03-17T19:04:58.3022700Z * [new branch] gh/dagitses/286/orig -> origin/gh/dagitses/286/orig 2022-03-17T19:04:58.3024637Z * [new branch] gh/dagitses/287/base -> origin/gh/dagitses/287/base 2022-03-17T19:04:58.3025830Z * [new branch] gh/dagitses/287/head -> origin/gh/dagitses/287/head 2022-03-17T19:04:58.3027165Z * [new branch] gh/dagitses/287/orig -> origin/gh/dagitses/287/orig 2022-03-17T19:04:58.3028929Z * [new branch] gh/dagitses/288/base -> origin/gh/dagitses/288/base 2022-03-17T19:04:58.3030254Z * [new branch] gh/dagitses/288/head -> origin/gh/dagitses/288/head 2022-03-17T19:04:58.3031643Z * [new branch] gh/dagitses/288/orig -> origin/gh/dagitses/288/orig 2022-03-17T19:04:58.3033393Z * [new branch] gh/dagitses/289/base -> origin/gh/dagitses/289/base 2022-03-17T19:04:58.3034682Z * [new branch] gh/dagitses/289/head -> origin/gh/dagitses/289/head 2022-03-17T19:04:58.3036027Z * [new branch] gh/dagitses/289/orig -> origin/gh/dagitses/289/orig 2022-03-17T19:04:58.3038015Z * [new branch] gh/dagitses/29/base -> origin/gh/dagitses/29/base 2022-03-17T19:04:58.3039110Z * [new branch] gh/dagitses/29/head -> origin/gh/dagitses/29/head 2022-03-17T19:04:58.3040519Z * [new branch] gh/dagitses/29/orig -> origin/gh/dagitses/29/orig 2022-03-17T19:04:58.3042151Z * [new branch] gh/dagitses/290/base -> origin/gh/dagitses/290/base 2022-03-17T19:04:58.3043664Z * [new branch] gh/dagitses/290/head -> origin/gh/dagitses/290/head 2022-03-17T19:04:58.3044947Z * [new branch] gh/dagitses/290/orig -> origin/gh/dagitses/290/orig 2022-03-17T19:04:58.3046704Z * [new branch] gh/dagitses/3/base -> origin/gh/dagitses/3/base 2022-03-17T19:04:58.3048081Z * [new branch] gh/dagitses/3/orig -> origin/gh/dagitses/3/orig 2022-03-17T19:04:58.3049894Z * [new branch] gh/dagitses/30/base -> origin/gh/dagitses/30/base 2022-03-17T19:04:58.3051212Z * [new branch] gh/dagitses/30/head -> origin/gh/dagitses/30/head 2022-03-17T19:04:58.3052520Z * [new branch] gh/dagitses/30/orig -> origin/gh/dagitses/30/orig 2022-03-17T19:04:58.3054290Z * [new branch] gh/dagitses/31/base -> origin/gh/dagitses/31/base 2022-03-17T19:04:58.3055658Z * [new branch] gh/dagitses/31/head -> origin/gh/dagitses/31/head 2022-03-17T19:04:58.3056981Z * [new branch] gh/dagitses/31/orig -> origin/gh/dagitses/31/orig 2022-03-17T19:04:58.3058813Z * [new branch] gh/dagitses/32/base -> origin/gh/dagitses/32/base 2022-03-17T19:04:58.3060232Z * [new branch] gh/dagitses/32/head -> origin/gh/dagitses/32/head 2022-03-17T19:04:58.3061662Z * [new branch] gh/dagitses/32/orig -> origin/gh/dagitses/32/orig 2022-03-17T19:04:58.3063308Z * [new branch] gh/dagitses/33/base -> origin/gh/dagitses/33/base 2022-03-17T19:04:58.3064693Z * [new branch] gh/dagitses/33/head -> origin/gh/dagitses/33/head 2022-03-17T19:04:58.3066044Z * [new branch] gh/dagitses/33/orig -> origin/gh/dagitses/33/orig 2022-03-17T19:04:58.3068020Z * [new branch] gh/dagitses/34/base -> origin/gh/dagitses/34/base 2022-03-17T19:04:58.3069439Z * [new branch] gh/dagitses/34/head -> origin/gh/dagitses/34/head 2022-03-17T19:04:58.3070830Z * [new branch] gh/dagitses/34/orig -> origin/gh/dagitses/34/orig 2022-03-17T19:04:58.3072444Z * [new branch] gh/dagitses/35/base -> origin/gh/dagitses/35/base 2022-03-17T19:04:58.3073894Z * [new branch] gh/dagitses/35/head -> origin/gh/dagitses/35/head 2022-03-17T19:04:58.3075289Z * [new branch] gh/dagitses/35/orig -> origin/gh/dagitses/35/orig 2022-03-17T19:04:58.3077324Z * [new branch] gh/dagitses/36/base -> origin/gh/dagitses/36/base 2022-03-17T19:04:58.3078688Z * [new branch] gh/dagitses/36/head -> origin/gh/dagitses/36/head 2022-03-17T19:04:58.3080055Z * [new branch] gh/dagitses/36/orig -> origin/gh/dagitses/36/orig 2022-03-17T19:04:58.3081809Z * [new branch] gh/dagitses/37/base -> origin/gh/dagitses/37/base 2022-03-17T19:04:58.3083450Z * [new branch] gh/dagitses/37/head -> origin/gh/dagitses/37/head 2022-03-17T19:04:58.3084759Z * [new branch] gh/dagitses/37/orig -> origin/gh/dagitses/37/orig 2022-03-17T19:04:58.3086452Z * [new branch] gh/dagitses/38/base -> origin/gh/dagitses/38/base 2022-03-17T19:04:58.3087909Z * [new branch] gh/dagitses/38/head -> origin/gh/dagitses/38/head 2022-03-17T19:04:58.3089346Z * [new branch] gh/dagitses/38/orig -> origin/gh/dagitses/38/orig 2022-03-17T19:04:58.3091004Z * [new branch] gh/dagitses/39/base -> origin/gh/dagitses/39/base 2022-03-17T19:04:58.3092296Z * [new branch] gh/dagitses/39/head -> origin/gh/dagitses/39/head 2022-03-17T19:04:58.3093502Z * [new branch] gh/dagitses/39/orig -> origin/gh/dagitses/39/orig 2022-03-17T19:04:58.3095298Z * [new branch] gh/dagitses/4/base -> origin/gh/dagitses/4/base 2022-03-17T19:04:58.3096681Z * [new branch] gh/dagitses/4/orig -> origin/gh/dagitses/4/orig 2022-03-17T19:04:58.3098635Z * [new branch] gh/dagitses/40/base -> origin/gh/dagitses/40/base 2022-03-17T19:04:58.3100054Z * [new branch] gh/dagitses/40/head -> origin/gh/dagitses/40/head 2022-03-17T19:04:58.3101383Z * [new branch] gh/dagitses/40/orig -> origin/gh/dagitses/40/orig 2022-03-17T19:04:58.3103167Z * [new branch] gh/dagitses/41/base -> origin/gh/dagitses/41/base 2022-03-17T19:04:58.3104517Z * [new branch] gh/dagitses/41/head -> origin/gh/dagitses/41/head 2022-03-17T19:04:58.3105869Z * [new branch] gh/dagitses/41/orig -> origin/gh/dagitses/41/orig 2022-03-17T19:04:58.3107606Z * [new branch] gh/dagitses/42/base -> origin/gh/dagitses/42/base 2022-03-17T19:04:58.3108900Z * [new branch] gh/dagitses/42/head -> origin/gh/dagitses/42/head 2022-03-17T19:04:58.3110232Z * [new branch] gh/dagitses/42/orig -> origin/gh/dagitses/42/orig 2022-03-17T19:04:58.3135238Z * [new branch] gh/dagitses/43/base -> origin/gh/dagitses/43/base 2022-03-17T19:04:58.3135953Z * [new branch] gh/dagitses/43/head -> origin/gh/dagitses/43/head 2022-03-17T19:04:58.3136479Z * [new branch] gh/dagitses/43/orig -> origin/gh/dagitses/43/orig 2022-03-17T19:04:58.3137181Z * [new branch] gh/dagitses/44/base -> origin/gh/dagitses/44/base 2022-03-17T19:04:58.3137869Z * [new branch] gh/dagitses/44/head -> origin/gh/dagitses/44/head 2022-03-17T19:04:58.3138535Z * [new branch] gh/dagitses/44/orig -> origin/gh/dagitses/44/orig 2022-03-17T19:04:58.3139187Z * [new branch] gh/dagitses/45/base -> origin/gh/dagitses/45/base 2022-03-17T19:04:58.3139826Z * [new branch] gh/dagitses/45/head -> origin/gh/dagitses/45/head 2022-03-17T19:04:58.3140467Z * [new branch] gh/dagitses/45/orig -> origin/gh/dagitses/45/orig 2022-03-17T19:04:58.3141093Z * [new branch] gh/dagitses/46/base -> origin/gh/dagitses/46/base 2022-03-17T19:04:58.3141786Z * [new branch] gh/dagitses/46/orig -> origin/gh/dagitses/46/orig 2022-03-17T19:04:58.3142416Z * [new branch] gh/dagitses/47/base -> origin/gh/dagitses/47/base 2022-03-17T19:04:58.3143076Z * [new branch] gh/dagitses/47/head -> origin/gh/dagitses/47/head 2022-03-17T19:04:58.3143946Z * [new branch] gh/dagitses/47/orig -> origin/gh/dagitses/47/orig 2022-03-17T19:04:58.3144702Z * [new branch] gh/dagitses/5/base -> origin/gh/dagitses/5/base 2022-03-17T19:04:58.3145378Z * [new branch] gh/dagitses/5/head -> origin/gh/dagitses/5/head 2022-03-17T19:04:58.3146027Z * [new branch] gh/dagitses/5/orig -> origin/gh/dagitses/5/orig 2022-03-17T19:04:58.3146654Z * [new branch] gh/dagitses/51/base -> origin/gh/dagitses/51/base 2022-03-17T19:04:58.3147297Z * [new branch] gh/dagitses/51/orig -> origin/gh/dagitses/51/orig 2022-03-17T19:04:58.3147945Z * [new branch] gh/dagitses/52/base -> origin/gh/dagitses/52/base 2022-03-17T19:04:58.3148603Z * [new branch] gh/dagitses/52/head -> origin/gh/dagitses/52/head 2022-03-17T19:04:58.3149239Z * [new branch] gh/dagitses/52/orig -> origin/gh/dagitses/52/orig 2022-03-17T19:04:58.3149943Z * [new branch] gh/dagitses/53/base -> origin/gh/dagitses/53/base 2022-03-17T19:04:58.3150766Z * [new branch] gh/dagitses/53/head -> origin/gh/dagitses/53/head 2022-03-17T19:04:58.3151469Z * [new branch] gh/dagitses/54/base -> origin/gh/dagitses/54/base 2022-03-17T19:04:58.3152138Z * [new branch] gh/dagitses/55/base -> origin/gh/dagitses/55/base 2022-03-17T19:04:58.3153423Z * [new branch] gh/dagitses/56/base -> origin/gh/dagitses/56/base 2022-03-17T19:04:58.3155061Z * [new branch] gh/dagitses/57/base -> origin/gh/dagitses/57/base 2022-03-17T19:04:58.3156860Z * [new branch] gh/dagitses/58/base -> origin/gh/dagitses/58/base 2022-03-17T19:04:58.3157929Z * [new branch] gh/dagitses/58/head -> origin/gh/dagitses/58/head 2022-03-17T19:04:58.3159663Z * [new branch] gh/dagitses/59/base -> origin/gh/dagitses/59/base 2022-03-17T19:04:58.3161218Z * [new branch] gh/dagitses/60/base -> origin/gh/dagitses/60/base 2022-03-17T19:04:58.3163333Z * [new branch] gh/dagitses/61/base -> origin/gh/dagitses/61/base 2022-03-17T19:04:58.3164827Z * [new branch] gh/dagitses/62/base -> origin/gh/dagitses/62/base 2022-03-17T19:04:58.3166631Z * [new branch] gh/dagitses/63/base -> origin/gh/dagitses/63/base 2022-03-17T19:04:58.3168005Z * [new branch] gh/dagitses/63/head -> origin/gh/dagitses/63/head 2022-03-17T19:04:58.3169885Z * [new branch] gh/dagitses/64/base -> origin/gh/dagitses/64/base 2022-03-17T19:04:58.3170910Z * [new branch] gh/dagitses/64/head -> origin/gh/dagitses/64/head 2022-03-17T19:04:58.3172304Z * [new branch] gh/dagitses/64/orig -> origin/gh/dagitses/64/orig 2022-03-17T19:04:58.3174331Z * [new branch] gh/dagitses/65/base -> origin/gh/dagitses/65/base 2022-03-17T19:04:58.3175678Z * [new branch] gh/dagitses/65/head -> origin/gh/dagitses/65/head 2022-03-17T19:04:58.3177158Z * [new branch] gh/dagitses/65/orig -> origin/gh/dagitses/65/orig 2022-03-17T19:04:58.3179012Z * [new branch] gh/dagitses/66/base -> origin/gh/dagitses/66/base 2022-03-17T19:04:58.3180176Z * [new branch] gh/dagitses/66/head -> origin/gh/dagitses/66/head 2022-03-17T19:04:58.3181569Z * [new branch] gh/dagitses/66/orig -> origin/gh/dagitses/66/orig 2022-03-17T19:04:58.3183462Z * [new branch] gh/dagitses/67/base -> origin/gh/dagitses/67/base 2022-03-17T19:04:58.3184791Z * [new branch] gh/dagitses/67/head -> origin/gh/dagitses/67/head 2022-03-17T19:04:58.3185977Z * [new branch] gh/dagitses/67/orig -> origin/gh/dagitses/67/orig 2022-03-17T19:04:58.3187911Z * [new branch] gh/dagitses/68/base -> origin/gh/dagitses/68/base 2022-03-17T19:04:58.3189437Z * [new branch] gh/dagitses/68/head -> origin/gh/dagitses/68/head 2022-03-17T19:04:58.3190639Z * [new branch] gh/dagitses/68/orig -> origin/gh/dagitses/68/orig 2022-03-17T19:04:58.3192393Z * [new branch] gh/dagitses/69/base -> origin/gh/dagitses/69/base 2022-03-17T19:04:58.3193688Z * [new branch] gh/dagitses/69/head -> origin/gh/dagitses/69/head 2022-03-17T19:04:58.3195059Z * [new branch] gh/dagitses/69/orig -> origin/gh/dagitses/69/orig 2022-03-17T19:04:58.3197000Z * [new branch] gh/dagitses/7/base -> origin/gh/dagitses/7/base 2022-03-17T19:04:58.3198096Z * [new branch] gh/dagitses/7/orig -> origin/gh/dagitses/7/orig 2022-03-17T19:04:58.3199862Z * [new branch] gh/dagitses/70/base -> origin/gh/dagitses/70/base 2022-03-17T19:04:58.3200958Z * [new branch] gh/dagitses/70/head -> origin/gh/dagitses/70/head 2022-03-17T19:04:58.3202409Z * [new branch] gh/dagitses/70/orig -> origin/gh/dagitses/70/orig 2022-03-17T19:04:58.3204189Z * [new branch] gh/dagitses/71/base -> origin/gh/dagitses/71/base 2022-03-17T19:04:58.3205264Z * [new branch] gh/dagitses/71/head -> origin/gh/dagitses/71/head 2022-03-17T19:04:58.3206611Z * [new branch] gh/dagitses/71/orig -> origin/gh/dagitses/71/orig 2022-03-17T19:04:58.3208548Z * [new branch] gh/dagitses/72/base -> origin/gh/dagitses/72/base 2022-03-17T19:04:58.3209707Z * [new branch] gh/dagitses/72/head -> origin/gh/dagitses/72/head 2022-03-17T19:04:58.3211028Z * [new branch] gh/dagitses/72/orig -> origin/gh/dagitses/72/orig 2022-03-17T19:04:58.3212873Z * [new branch] gh/dagitses/73/base -> origin/gh/dagitses/73/base 2022-03-17T19:04:58.3214021Z * [new branch] gh/dagitses/73/head -> origin/gh/dagitses/73/head 2022-03-17T19:04:58.3227695Z * [new branch] gh/dagitses/73/orig -> origin/gh/dagitses/73/orig 2022-03-17T19:04:58.3228356Z * [new branch] gh/dagitses/74/base -> origin/gh/dagitses/74/base 2022-03-17T19:04:58.3229023Z * [new branch] gh/dagitses/74/head -> origin/gh/dagitses/74/head 2022-03-17T19:04:58.3229628Z * [new branch] gh/dagitses/74/orig -> origin/gh/dagitses/74/orig 2022-03-17T19:04:58.3230191Z * [new branch] gh/dagitses/75/base -> origin/gh/dagitses/75/base 2022-03-17T19:04:58.3230753Z * [new branch] gh/dagitses/75/head -> origin/gh/dagitses/75/head 2022-03-17T19:04:58.3231312Z * [new branch] gh/dagitses/75/orig -> origin/gh/dagitses/75/orig 2022-03-17T19:04:58.3231855Z * [new branch] gh/dagitses/76/base -> origin/gh/dagitses/76/base 2022-03-17T19:04:58.3232421Z * [new branch] gh/dagitses/76/head -> origin/gh/dagitses/76/head 2022-03-17T19:04:58.3232988Z * [new branch] gh/dagitses/76/orig -> origin/gh/dagitses/76/orig 2022-03-17T19:04:58.3233554Z * [new branch] gh/dagitses/77/base -> origin/gh/dagitses/77/base 2022-03-17T19:04:58.3234108Z * [new branch] gh/dagitses/77/head -> origin/gh/dagitses/77/head 2022-03-17T19:04:58.3234665Z * [new branch] gh/dagitses/77/orig -> origin/gh/dagitses/77/orig 2022-03-17T19:04:58.3235207Z * [new branch] gh/dagitses/78/base -> origin/gh/dagitses/78/base 2022-03-17T19:04:58.3236314Z * [new branch] gh/dagitses/78/orig -> origin/gh/dagitses/78/orig 2022-03-17T19:04:58.3268902Z * [new branch] gh/dagitses/79/base -> origin/gh/dagitses/79/base 2022-03-17T19:04:58.3269524Z * [new branch] gh/dagitses/79/head -> origin/gh/dagitses/79/head 2022-03-17T19:04:58.3270233Z * [new branch] gh/dagitses/79/orig -> origin/gh/dagitses/79/orig 2022-03-17T19:04:58.3271340Z * [new branch] gh/dagitses/8/base -> origin/gh/dagitses/8/base 2022-03-17T19:04:58.3272253Z * [new branch] gh/dagitses/8/orig -> origin/gh/dagitses/8/orig 2022-03-17T19:04:58.3273130Z * [new branch] gh/dagitses/80/base -> origin/gh/dagitses/80/base 2022-03-17T19:04:58.3273878Z * [new branch] gh/dagitses/80/head -> origin/gh/dagitses/80/head 2022-03-17T19:04:58.3274465Z * [new branch] gh/dagitses/80/orig -> origin/gh/dagitses/80/orig 2022-03-17T19:04:58.3275288Z * [new branch] gh/dagitses/81/base -> origin/gh/dagitses/81/base 2022-03-17T19:04:58.3276023Z * [new branch] gh/dagitses/81/head -> origin/gh/dagitses/81/head 2022-03-17T19:04:58.3276824Z * [new branch] gh/dagitses/82/base -> origin/gh/dagitses/82/base 2022-03-17T19:04:58.3277459Z * [new branch] gh/dagitses/82/head -> origin/gh/dagitses/82/head 2022-03-17T19:04:58.3278285Z * [new branch] gh/dagitses/83/base -> origin/gh/dagitses/83/base 2022-03-17T19:04:58.3279678Z * [new branch] gh/dagitses/83/head -> origin/gh/dagitses/83/head 2022-03-17T19:04:58.3280614Z * [new branch] gh/dagitses/84/base -> origin/gh/dagitses/84/base 2022-03-17T19:04:58.3281629Z * [new branch] gh/dagitses/84/head -> origin/gh/dagitses/84/head 2022-03-17T19:04:58.3282879Z * [new branch] gh/dagitses/85/base -> origin/gh/dagitses/85/base 2022-03-17T19:04:58.3283651Z * [new branch] gh/dagitses/85/head -> origin/gh/dagitses/85/head 2022-03-17T19:04:58.3284550Z * [new branch] gh/dagitses/86/base -> origin/gh/dagitses/86/base 2022-03-17T19:04:58.3285606Z * [new branch] gh/dagitses/86/head -> origin/gh/dagitses/86/head 2022-03-17T19:04:58.3286603Z * [new branch] gh/dagitses/87/base -> origin/gh/dagitses/87/base 2022-03-17T19:04:58.3287646Z * [new branch] gh/dagitses/87/head -> origin/gh/dagitses/87/head 2022-03-17T19:04:58.3288682Z * [new branch] gh/dagitses/88/base -> origin/gh/dagitses/88/base 2022-03-17T19:04:58.3289574Z * [new branch] gh/dagitses/88/head -> origin/gh/dagitses/88/head 2022-03-17T19:04:58.3290603Z * [new branch] gh/dagitses/89/base -> origin/gh/dagitses/89/base 2022-03-17T19:04:58.3291621Z * [new branch] gh/dagitses/89/head -> origin/gh/dagitses/89/head 2022-03-17T19:04:58.3292591Z * [new branch] gh/dagitses/9/base -> origin/gh/dagitses/9/base 2022-03-17T19:04:58.3293656Z * [new branch] gh/dagitses/9/orig -> origin/gh/dagitses/9/orig 2022-03-17T19:04:58.3294725Z * [new branch] gh/dagitses/90/base -> origin/gh/dagitses/90/base 2022-03-17T19:04:58.3295609Z * [new branch] gh/dagitses/90/head -> origin/gh/dagitses/90/head 2022-03-17T19:04:58.3296611Z * [new branch] gh/dagitses/91/base -> origin/gh/dagitses/91/base 2022-03-17T19:04:58.3297688Z * [new branch] gh/dagitses/91/head -> origin/gh/dagitses/91/head 2022-03-17T19:04:58.3298623Z * [new branch] gh/dagitses/92/base -> origin/gh/dagitses/92/base 2022-03-17T19:04:58.3299617Z * [new branch] gh/dagitses/92/head -> origin/gh/dagitses/92/head 2022-03-17T19:04:58.3300648Z * [new branch] gh/dagitses/93/base -> origin/gh/dagitses/93/base 2022-03-17T19:04:58.3301595Z * [new branch] gh/dagitses/93/head -> origin/gh/dagitses/93/head 2022-03-17T19:04:58.3302615Z * [new branch] gh/dagitses/94/base -> origin/gh/dagitses/94/base 2022-03-17T19:04:58.3303609Z * [new branch] gh/dagitses/94/head -> origin/gh/dagitses/94/head 2022-03-17T19:04:58.3304639Z * [new branch] gh/dagitses/95/base -> origin/gh/dagitses/95/base 2022-03-17T19:04:58.3306001Z * [new branch] gh/dagitses/95/head -> origin/gh/dagitses/95/head 2022-03-17T19:04:58.3306921Z * [new branch] gh/dagitses/96/base -> origin/gh/dagitses/96/base 2022-03-17T19:04:58.3307867Z * [new branch] gh/dagitses/96/head -> origin/gh/dagitses/96/head 2022-03-17T19:04:58.3308900Z * [new branch] gh/dagitses/97/base -> origin/gh/dagitses/97/base 2022-03-17T19:04:58.3309899Z * [new branch] gh/dagitses/97/head -> origin/gh/dagitses/97/head 2022-03-17T19:04:58.3310961Z * [new branch] gh/dagitses/98/base -> origin/gh/dagitses/98/base 2022-03-17T19:04:58.3311880Z * [new branch] gh/dagitses/98/head -> origin/gh/dagitses/98/head 2022-03-17T19:04:58.3312918Z * [new branch] gh/dagitses/99/base -> origin/gh/dagitses/99/base 2022-03-17T19:04:58.3313943Z * [new branch] gh/dagitses/99/head -> origin/gh/dagitses/99/head 2022-03-17T19:04:58.3315025Z * [new branch] gh/davidberard98/2/base -> origin/gh/davidberard98/2/base 2022-03-17T19:04:58.3316539Z * [new branch] gh/davidberard98/2/head -> origin/gh/davidberard98/2/head 2022-03-17T19:04:58.3317752Z * [new branch] gh/davidberard98/2/orig -> origin/gh/davidberard98/2/orig 2022-03-17T19:04:58.3318792Z * [new branch] gh/davidberard98/23/base -> origin/gh/davidberard98/23/base 2022-03-17T19:04:58.3319780Z * [new branch] gh/davidberard98/23/head -> origin/gh/davidberard98/23/head 2022-03-17T19:04:58.3320782Z * [new branch] gh/davidberard98/23/orig -> origin/gh/davidberard98/23/orig 2022-03-17T19:04:58.3321818Z * [new branch] gh/davidberard98/28/base -> origin/gh/davidberard98/28/base 2022-03-17T19:04:58.3323293Z * [new branch] gh/davidberard98/28/orig -> origin/gh/davidberard98/28/orig 2022-03-17T19:04:58.3324338Z * [new branch] gh/davidberard98/30/base -> origin/gh/davidberard98/30/base 2022-03-17T19:04:58.3325424Z * [new branch] gh/davidberard98/30/head -> origin/gh/davidberard98/30/head 2022-03-17T19:04:58.3326483Z * [new branch] gh/davidberard98/30/orig -> origin/gh/davidberard98/30/orig 2022-03-17T19:04:58.3327559Z * [new branch] gh/davidberard98/31/base -> origin/gh/davidberard98/31/base 2022-03-17T19:04:58.3328506Z * [new branch] gh/davidberard98/31/head -> origin/gh/davidberard98/31/head 2022-03-17T19:04:58.3329288Z * [new branch] gh/davidberard98/31/orig -> origin/gh/davidberard98/31/orig 2022-03-17T19:04:58.3330099Z * [new branch] gh/davidberard98/34/base -> origin/gh/davidberard98/34/base 2022-03-17T19:04:58.3330841Z * [new branch] gh/davidberard98/34/head -> origin/gh/davidberard98/34/head 2022-03-17T19:04:58.3332144Z * [new branch] gh/davidberard98/34/orig -> origin/gh/davidberard98/34/orig 2022-03-17T19:04:58.3333890Z * [new branch] gh/davidberard98/38/base -> origin/gh/davidberard98/38/base 2022-03-17T19:04:58.3335176Z * [new branch] gh/davidberard98/38/head -> origin/gh/davidberard98/38/head 2022-03-17T19:04:58.3336526Z * [new branch] gh/davidberard98/38/orig -> origin/gh/davidberard98/38/orig 2022-03-17T19:04:58.3338701Z * [new branch] gh/davidberard98/47/base -> origin/gh/davidberard98/47/base 2022-03-17T19:04:58.3340192Z * [new branch] gh/davidberard98/47/head -> origin/gh/davidberard98/47/head 2022-03-17T19:04:58.3341682Z * [new branch] gh/davidberard98/47/orig -> origin/gh/davidberard98/47/orig 2022-03-17T19:04:58.3343997Z * [new branch] gh/davidberard98/49/base -> origin/gh/davidberard98/49/base 2022-03-17T19:04:58.3345348Z * [new branch] gh/davidberard98/49/head -> origin/gh/davidberard98/49/head 2022-03-17T19:04:58.3346781Z * [new branch] gh/davidberard98/49/orig -> origin/gh/davidberard98/49/orig 2022-03-17T19:04:58.3349184Z * [new branch] gh/davidberard98/52/base -> origin/gh/davidberard98/52/base 2022-03-17T19:04:58.3350401Z * [new branch] gh/davidberard98/52/head -> origin/gh/davidberard98/52/head 2022-03-17T19:04:58.3351912Z * [new branch] gh/davidberard98/52/orig -> origin/gh/davidberard98/52/orig 2022-03-17T19:04:58.3354028Z * [new branch] gh/davidberard98/53/base -> origin/gh/davidberard98/53/base 2022-03-17T19:04:58.3355300Z * [new branch] gh/davidberard98/53/head -> origin/gh/davidberard98/53/head 2022-03-17T19:04:58.3356709Z * [new branch] gh/davidberard98/53/orig -> origin/gh/davidberard98/53/orig 2022-03-17T19:04:58.3358611Z * [new branch] gh/davidberard98/54/base -> origin/gh/davidberard98/54/base 2022-03-17T19:04:58.3360122Z * [new branch] gh/davidberard98/54/head -> origin/gh/davidberard98/54/head 2022-03-17T19:04:58.3361446Z * [new branch] gh/davidberard98/54/orig -> origin/gh/davidberard98/54/orig 2022-03-17T19:04:58.3363660Z * [new branch] gh/davidberard98/56/base -> origin/gh/davidberard98/56/base 2022-03-17T19:04:58.3365047Z * [new branch] gh/davidberard98/56/head -> origin/gh/davidberard98/56/head 2022-03-17T19:04:58.3366382Z * [new branch] gh/davidberard98/56/orig -> origin/gh/davidberard98/56/orig 2022-03-17T19:04:58.3368548Z * [new branch] gh/davidberard98/58/base -> origin/gh/davidberard98/58/base 2022-03-17T19:04:58.3369664Z * [new branch] gh/davidberard98/58/head -> origin/gh/davidberard98/58/head 2022-03-17T19:04:58.3370992Z * [new branch] gh/davidberard98/58/orig -> origin/gh/davidberard98/58/orig 2022-03-17T19:04:58.3372638Z * [new branch] gh/davidberard98/59/base -> origin/gh/davidberard98/59/base 2022-03-17T19:04:58.3374025Z * [new branch] gh/davidberard98/59/head -> origin/gh/davidberard98/59/head 2022-03-17T19:04:58.3375405Z * [new branch] gh/davidberard98/59/orig -> origin/gh/davidberard98/59/orig 2022-03-17T19:04:58.3377064Z * [new branch] gh/davidberard98/61/base -> origin/gh/davidberard98/61/base 2022-03-17T19:04:58.3378361Z * [new branch] gh/davidberard98/61/head -> origin/gh/davidberard98/61/head 2022-03-17T19:04:58.3379679Z * [new branch] gh/davidberard98/61/orig -> origin/gh/davidberard98/61/orig 2022-03-17T19:04:58.3381314Z * [new branch] gh/davidberard98/63/base -> origin/gh/davidberard98/63/base 2022-03-17T19:04:58.3382668Z * [new branch] gh/davidberard98/63/head -> origin/gh/davidberard98/63/head 2022-03-17T19:04:58.3383984Z * [new branch] gh/davidberard98/63/orig -> origin/gh/davidberard98/63/orig 2022-03-17T19:04:58.3385659Z * [new branch] gh/davidberard98/64/base -> origin/gh/davidberard98/64/base 2022-03-17T19:04:58.3387171Z * [new branch] gh/davidberard98/64/head -> origin/gh/davidberard98/64/head 2022-03-17T19:04:58.3388495Z * [new branch] gh/davidberard98/64/orig -> origin/gh/davidberard98/64/orig 2022-03-17T19:04:58.3390015Z * [new branch] gh/davidberard98/65/base -> origin/gh/davidberard98/65/base 2022-03-17T19:04:58.3391394Z * [new branch] gh/davidberard98/65/head -> origin/gh/davidberard98/65/head 2022-03-17T19:04:58.3392950Z * [new branch] gh/davidberard98/65/orig -> origin/gh/davidberard98/65/orig 2022-03-17T19:04:58.3394806Z * [new branch] gh/dbort/1/base -> origin/gh/dbort/1/base 2022-03-17T19:04:58.3396080Z * [new branch] gh/dbort/1/head -> origin/gh/dbort/1/head 2022-03-17T19:04:58.3397412Z * [new branch] gh/dbort/1/orig -> origin/gh/dbort/1/orig 2022-03-17T19:04:58.3399159Z * [new branch] gh/dbort/2/base -> origin/gh/dbort/2/base 2022-03-17T19:04:58.3400432Z * [new branch] gh/dbort/2/head -> origin/gh/dbort/2/head 2022-03-17T19:04:58.3402224Z * [new branch] gh/dbort/2/orig -> origin/gh/dbort/2/orig 2022-03-17T19:04:58.3404091Z * [new branch] gh/desertfire/18/base -> origin/gh/desertfire/18/base 2022-03-17T19:04:58.3405340Z * [new branch] gh/desertfire/18/head -> origin/gh/desertfire/18/head 2022-03-17T19:04:58.3406685Z * [new branch] gh/desertfire/18/orig -> origin/gh/desertfire/18/orig 2022-03-17T19:04:58.3408570Z * [new branch] gh/desertfire/20/base -> origin/gh/desertfire/20/base 2022-03-17T19:04:58.3409315Z * [new branch] gh/desertfire/20/head -> origin/gh/desertfire/20/head 2022-03-17T19:04:58.3410643Z * [new branch] gh/desertfire/20/orig -> origin/gh/desertfire/20/orig 2022-03-17T19:04:58.3412548Z * [new branch] gh/dhruvbird/20/base -> origin/gh/dhruvbird/20/base 2022-03-17T19:04:58.3413941Z * [new branch] gh/dhruvbird/20/head -> origin/gh/dhruvbird/20/head 2022-03-17T19:04:58.3415332Z * [new branch] gh/dhruvbird/20/orig -> origin/gh/dhruvbird/20/orig 2022-03-17T19:04:58.3417390Z * [new branch] gh/dhruvbird/23/base -> origin/gh/dhruvbird/23/base 2022-03-17T19:04:58.3418910Z * [new branch] gh/dhruvbird/23/head -> origin/gh/dhruvbird/23/head 2022-03-17T19:04:58.3421026Z * [new branch] gh/dhruvbird/23/orig -> origin/gh/dhruvbird/23/orig 2022-03-17T19:04:58.3422163Z * [new branch] gh/dhruvbird/31/base -> origin/gh/dhruvbird/31/base 2022-03-17T19:04:58.3423570Z * [new branch] gh/dhruvbird/31/head -> origin/gh/dhruvbird/31/head 2022-03-17T19:04:58.3424772Z * [new branch] gh/dhruvbird/31/orig -> origin/gh/dhruvbird/31/orig 2022-03-17T19:04:58.3426523Z * [new branch] gh/dhruvbird/52/base -> origin/gh/dhruvbird/52/base 2022-03-17T19:04:58.3427906Z * [new branch] gh/dhruvbird/52/head -> origin/gh/dhruvbird/52/head 2022-03-17T19:04:58.3429087Z * [new branch] gh/dhruvbird/52/orig -> origin/gh/dhruvbird/52/orig 2022-03-17T19:04:58.3430921Z * [new branch] gh/dhruvbird/55/base -> origin/gh/dhruvbird/55/base 2022-03-17T19:04:58.3432446Z * [new branch] gh/dhruvbird/55/head -> origin/gh/dhruvbird/55/head 2022-03-17T19:04:58.3434056Z * [new branch] gh/dhruvbird/55/orig -> origin/gh/dhruvbird/55/orig 2022-03-17T19:04:58.3435738Z * [new branch] gh/dhruvbird/56/base -> origin/gh/dhruvbird/56/base 2022-03-17T19:04:58.3437091Z * [new branch] gh/dhruvbird/56/head -> origin/gh/dhruvbird/56/head 2022-03-17T19:04:58.3438636Z * [new branch] gh/dhruvbird/57/base -> origin/gh/dhruvbird/57/base 2022-03-17T19:04:58.3439846Z * [new branch] gh/dhruvbird/57/head -> origin/gh/dhruvbird/57/head 2022-03-17T19:04:58.3441384Z * [new branch] gh/dhruvbird/58/base -> origin/gh/dhruvbird/58/base 2022-03-17T19:04:58.3443142Z * [new branch] gh/dhruvbird/58/head -> origin/gh/dhruvbird/58/head 2022-03-17T19:04:58.3444956Z * [new branch] gh/dhruvbird/59/base -> origin/gh/dhruvbird/59/base 2022-03-17T19:04:58.3446594Z * [new branch] gh/dhruvbird/59/head -> origin/gh/dhruvbird/59/head 2022-03-17T19:04:58.3448018Z * [new branch] gh/dhruvbird/60/base -> origin/gh/dhruvbird/60/base 2022-03-17T19:04:58.3449204Z * [new branch] gh/dhruvbird/60/head -> origin/gh/dhruvbird/60/head 2022-03-17T19:04:58.3450760Z * [new branch] gh/dhruvbird/61/base -> origin/gh/dhruvbird/61/base 2022-03-17T19:04:58.3452143Z * [new branch] gh/dhruvbird/61/head -> origin/gh/dhruvbird/61/head 2022-03-17T19:04:58.3453732Z * [new branch] gh/dhruvbird/66/base -> origin/gh/dhruvbird/66/base 2022-03-17T19:04:58.3455365Z * [new branch] gh/dhruvbird/66/head -> origin/gh/dhruvbird/66/head 2022-03-17T19:04:58.3456515Z * [new branch] gh/dhruvbird/66/orig -> origin/gh/dhruvbird/66/orig 2022-03-17T19:04:58.3458385Z * [new branch] gh/dhruvbird/75/base -> origin/gh/dhruvbird/75/base 2022-03-17T19:04:58.3459788Z * [new branch] gh/dhruvbird/75/head -> origin/gh/dhruvbird/75/head 2022-03-17T19:04:58.3461296Z * [new branch] gh/dhruvbird/75/orig -> origin/gh/dhruvbird/75/orig 2022-03-17T19:04:58.3462951Z * [new branch] gh/dhruvbird/87/base -> origin/gh/dhruvbird/87/base 2022-03-17T19:04:58.3464332Z * [new branch] gh/dhruvbird/87/head -> origin/gh/dhruvbird/87/head 2022-03-17T19:04:58.3465840Z * [new branch] gh/dhruvbird/88/base -> origin/gh/dhruvbird/88/base 2022-03-17T19:04:58.3467157Z * [new branch] gh/dhruvbird/88/head -> origin/gh/dhruvbird/88/head 2022-03-17T19:04:58.3469208Z * [new branch] gh/dhruvbird/92/base -> origin/gh/dhruvbird/92/base 2022-03-17T19:04:58.3470525Z * [new branch] gh/dhruvbird/92/head -> origin/gh/dhruvbird/92/head 2022-03-17T19:04:58.3471993Z * [new branch] gh/dhruvbird/92/orig -> origin/gh/dhruvbird/92/orig 2022-03-17T19:04:58.3473681Z * [new branch] gh/dhruvbird/93/base -> origin/gh/dhruvbird/93/base 2022-03-17T19:04:58.3475043Z * [new branch] gh/dhruvbird/93/head -> origin/gh/dhruvbird/93/head 2022-03-17T19:04:58.3476247Z * [new branch] gh/dhruvbird/93/orig -> origin/gh/dhruvbird/93/orig 2022-03-17T19:04:58.3477896Z * [new branch] gh/dhruvbird/94/base -> origin/gh/dhruvbird/94/base 2022-03-17T19:04:58.3479074Z * [new branch] gh/dhruvbird/94/head -> origin/gh/dhruvbird/94/head 2022-03-17T19:04:58.3480375Z * [new branch] gh/dhruvbird/94/orig -> origin/gh/dhruvbird/94/orig 2022-03-17T19:04:58.3482396Z * [new branch] gh/dhruvbird/95/base -> origin/gh/dhruvbird/95/base 2022-03-17T19:04:58.3483870Z * [new branch] gh/dhruvbird/95/head -> origin/gh/dhruvbird/95/head 2022-03-17T19:04:58.3485229Z * [new branch] gh/dhruvbird/95/orig -> origin/gh/dhruvbird/95/orig 2022-03-17T19:04:58.3487018Z * [new branch] gh/dhruvbird/96/base -> origin/gh/dhruvbird/96/base 2022-03-17T19:04:58.3488366Z * [new branch] gh/dhruvbird/96/head -> origin/gh/dhruvbird/96/head 2022-03-17T19:04:58.3489683Z * [new branch] gh/dhruvbird/96/orig -> origin/gh/dhruvbird/96/orig 2022-03-17T19:04:58.3491425Z * [new branch] gh/dhruvbird/97/base -> origin/gh/dhruvbird/97/base 2022-03-17T19:04:58.3492724Z * [new branch] gh/dhruvbird/97/head -> origin/gh/dhruvbird/97/head 2022-03-17T19:04:58.3494098Z * [new branch] gh/dhruvbird/97/orig -> origin/gh/dhruvbird/97/orig 2022-03-17T19:04:58.3496081Z * [new branch] gh/dracifer/1/base -> origin/gh/dracifer/1/base 2022-03-17T19:04:58.3497270Z * [new branch] gh/dracifer/1/head -> origin/gh/dracifer/1/head 2022-03-17T19:04:58.3498662Z * [new branch] gh/dracifer/1/orig -> origin/gh/dracifer/1/orig 2022-03-17T19:04:58.3500230Z * [new branch] gh/dracifer/2/base -> origin/gh/dracifer/2/base 2022-03-17T19:04:58.3501782Z * [new branch] gh/dracifer/2/orig -> origin/gh/dracifer/2/orig 2022-03-17T19:04:58.3503647Z * [new branch] gh/drdarshan/1/base -> origin/gh/drdarshan/1/base 2022-03-17T19:04:58.3505030Z * [new branch] gh/drdarshan/1/head -> origin/gh/drdarshan/1/head 2022-03-17T19:04:58.3506209Z * [new branch] gh/drdarshan/1/orig -> origin/gh/drdarshan/1/orig 2022-03-17T19:04:58.3508041Z * [new branch] gh/drdarshan/2/base -> origin/gh/drdarshan/2/base 2022-03-17T19:04:58.3509470Z * [new branch] gh/drdarshan/2/orig -> origin/gh/drdarshan/2/orig 2022-03-17T19:04:58.3511074Z * [new branch] gh/drdarshan/3/base -> origin/gh/drdarshan/3/base 2022-03-17T19:04:58.3512255Z * [new branch] gh/drdarshan/3/orig -> origin/gh/drdarshan/3/orig 2022-03-17T19:04:58.3514170Z * [new branch] gh/dreiss/1/base -> origin/gh/dreiss/1/base 2022-03-17T19:04:58.3515545Z * [new branch] gh/dreiss/1/head -> origin/gh/dreiss/1/head 2022-03-17T19:04:58.3516708Z * [new branch] gh/dreiss/1/orig -> origin/gh/dreiss/1/orig 2022-03-17T19:04:58.3518599Z * [new branch] gh/dreiss/10/base -> origin/gh/dreiss/10/base 2022-03-17T19:04:58.3519980Z * [new branch] gh/dreiss/10/head -> origin/gh/dreiss/10/head 2022-03-17T19:04:58.3521409Z * [new branch] gh/dreiss/10/orig -> origin/gh/dreiss/10/orig 2022-03-17T19:04:58.3523432Z * [new branch] gh/dreiss/105/base -> origin/gh/dreiss/105/base 2022-03-17T19:04:58.3524758Z * [new branch] gh/dreiss/105/head -> origin/gh/dreiss/105/head 2022-03-17T19:04:58.3526137Z * [new branch] gh/dreiss/105/orig -> origin/gh/dreiss/105/orig 2022-03-17T19:04:58.3527913Z * [new branch] gh/dreiss/106/base -> origin/gh/dreiss/106/base 2022-03-17T19:04:58.3529315Z * [new branch] gh/dreiss/106/head -> origin/gh/dreiss/106/head 2022-03-17T19:04:58.3532156Z * [new branch] gh/dreiss/106/orig -> origin/gh/dreiss/106/orig 2022-03-17T19:04:58.3536895Z * [new branch] gh/dreiss/109/base -> origin/gh/dreiss/109/base 2022-03-17T19:04:58.3540731Z * [new branch] gh/dreiss/109/head -> origin/gh/dreiss/109/head 2022-03-17T19:04:58.3544284Z * [new branch] gh/dreiss/109/orig -> origin/gh/dreiss/109/orig 2022-03-17T19:04:58.3548706Z * [new branch] gh/dreiss/110/base -> origin/gh/dreiss/110/base 2022-03-17T19:04:58.3552140Z * [new branch] gh/dreiss/110/head -> origin/gh/dreiss/110/head 2022-03-17T19:04:58.3555624Z * [new branch] gh/dreiss/110/orig -> origin/gh/dreiss/110/orig 2022-03-17T19:04:58.3559471Z * [new branch] gh/dreiss/2/base -> origin/gh/dreiss/2/base 2022-03-17T19:04:58.3563805Z * [new branch] gh/dreiss/2/head -> origin/gh/dreiss/2/head 2022-03-17T19:04:58.3567633Z * [new branch] gh/dreiss/2/orig -> origin/gh/dreiss/2/orig 2022-03-17T19:04:58.3570926Z * [new branch] gh/dreiss/25/base -> origin/gh/dreiss/25/base 2022-03-17T19:04:58.3572117Z * [new branch] gh/dreiss/25/head -> origin/gh/dreiss/25/head 2022-03-17T19:04:58.3573546Z * [new branch] gh/dreiss/25/orig -> origin/gh/dreiss/25/orig 2022-03-17T19:04:58.3575442Z * [new branch] gh/dreiss/28/base -> origin/gh/dreiss/28/base 2022-03-17T19:04:58.3576915Z * [new branch] gh/dreiss/28/head -> origin/gh/dreiss/28/head 2022-03-17T19:04:58.3578278Z * [new branch] gh/dreiss/28/orig -> origin/gh/dreiss/28/orig 2022-03-17T19:04:58.3580195Z * [new branch] gh/dreiss/33/base -> origin/gh/dreiss/33/base 2022-03-17T19:04:58.3581611Z * [new branch] gh/dreiss/33/head -> origin/gh/dreiss/33/head 2022-03-17T19:04:58.3582944Z * [new branch] gh/dreiss/33/orig -> origin/gh/dreiss/33/orig 2022-03-17T19:04:58.3584585Z * [new branch] gh/dreiss/36/base -> origin/gh/dreiss/36/base 2022-03-17T19:04:58.3585687Z * [new branch] gh/dreiss/36/head -> origin/gh/dreiss/36/head 2022-03-17T19:04:58.3587163Z * [new branch] gh/dreiss/36/orig -> origin/gh/dreiss/36/orig 2022-03-17T19:04:58.3588757Z * [new branch] gh/dreiss/39/base -> origin/gh/dreiss/39/base 2022-03-17T19:04:58.3590418Z * [new branch] gh/dreiss/39/head -> origin/gh/dreiss/39/head 2022-03-17T19:04:58.3591461Z * [new branch] gh/dreiss/39/orig -> origin/gh/dreiss/39/orig 2022-03-17T19:04:58.3593214Z * [new branch] gh/dreiss/43/base -> origin/gh/dreiss/43/base 2022-03-17T19:04:58.3594575Z * [new branch] gh/dreiss/43/head -> origin/gh/dreiss/43/head 2022-03-17T19:04:58.3595880Z * [new branch] gh/dreiss/43/orig -> origin/gh/dreiss/43/orig 2022-03-17T19:04:58.3597706Z * [new branch] gh/dreiss/63/base -> origin/gh/dreiss/63/base 2022-03-17T19:04:58.3599096Z * [new branch] gh/dreiss/63/head -> origin/gh/dreiss/63/head 2022-03-17T19:04:58.3600522Z * [new branch] gh/dreiss/63/orig -> origin/gh/dreiss/63/orig 2022-03-17T19:04:58.3602294Z * [new branch] gh/dreiss/67/base -> origin/gh/dreiss/67/base 2022-03-17T19:04:58.3604084Z * [new branch] gh/dreiss/67/head -> origin/gh/dreiss/67/head 2022-03-17T19:04:58.3605481Z * [new branch] gh/dreiss/67/orig -> origin/gh/dreiss/67/orig 2022-03-17T19:04:58.3606994Z * [new branch] gh/dreiss/68/base -> origin/gh/dreiss/68/base 2022-03-17T19:04:58.3608301Z * [new branch] gh/dreiss/68/head -> origin/gh/dreiss/68/head 2022-03-17T19:04:58.3609639Z * [new branch] gh/dreiss/68/orig -> origin/gh/dreiss/68/orig 2022-03-17T19:04:58.3611348Z * [new branch] gh/dreiss/71/base -> origin/gh/dreiss/71/base 2022-03-17T19:04:58.3612752Z * [new branch] gh/dreiss/71/head -> origin/gh/dreiss/71/head 2022-03-17T19:04:58.3614088Z * [new branch] gh/dreiss/71/orig -> origin/gh/dreiss/71/orig 2022-03-17T19:04:58.3615625Z * [new branch] gh/dreiss/72/base -> origin/gh/dreiss/72/base 2022-03-17T19:04:58.3616992Z * [new branch] gh/dreiss/72/head -> origin/gh/dreiss/72/head 2022-03-17T19:04:58.3618332Z * [new branch] gh/dreiss/72/orig -> origin/gh/dreiss/72/orig 2022-03-17T19:04:58.3619861Z * [new branch] gh/dreiss/73/base -> origin/gh/dreiss/73/base 2022-03-17T19:04:58.3621236Z * [new branch] gh/dreiss/73/head -> origin/gh/dreiss/73/head 2022-03-17T19:04:58.3626906Z * [new branch] gh/dreiss/73/orig -> origin/gh/dreiss/73/orig 2022-03-17T19:04:58.3627822Z * [new branch] gh/dreiss/75/base -> origin/gh/dreiss/75/base 2022-03-17T19:04:58.3628595Z * [new branch] gh/dreiss/75/head -> origin/gh/dreiss/75/head 2022-03-17T19:04:58.3629264Z * [new branch] gh/dreiss/80/base -> origin/gh/dreiss/80/base 2022-03-17T19:04:58.3630015Z * [new branch] gh/dreiss/80/head -> origin/gh/dreiss/80/head 2022-03-17T19:04:58.3630669Z * [new branch] gh/dreiss/80/orig -> origin/gh/dreiss/80/orig 2022-03-17T19:04:58.3633274Z * [new branch] gh/dskhudia/20/base -> origin/gh/dskhudia/20/base 2022-03-17T19:04:58.3633973Z * [new branch] gh/dskhudia/20/head -> origin/gh/dskhudia/20/head 2022-03-17T19:04:58.3635075Z * [new branch] gh/dskhudia/21/base -> origin/gh/dskhudia/21/base 2022-03-17T19:04:58.3636396Z * [new branch] gh/dskhudia/21/head -> origin/gh/dskhudia/21/head 2022-03-17T19:04:58.3637888Z * [new branch] gh/dskhudia/22/base -> origin/gh/dskhudia/22/base 2022-03-17T19:04:58.3639108Z * [new branch] gh/dskhudia/22/head -> origin/gh/dskhudia/22/head 2022-03-17T19:04:58.3640642Z * [new branch] gh/dskhudia/23/base -> origin/gh/dskhudia/23/base 2022-03-17T19:04:58.3642032Z * [new branch] gh/dskhudia/23/head -> origin/gh/dskhudia/23/head 2022-03-17T19:04:58.3644208Z * [new branch] gh/dskhudia/24/base -> origin/gh/dskhudia/24/base 2022-03-17T19:04:58.3645513Z * [new branch] gh/dskhudia/24/head -> origin/gh/dskhudia/24/head 2022-03-17T19:04:58.3650593Z * [new branch] gh/dskhudia/27/base -> origin/gh/dskhudia/27/base 2022-03-17T19:04:58.3651423Z * [new branch] gh/dskhudia/27/head -> origin/gh/dskhudia/27/head 2022-03-17T19:04:58.3652189Z * [new branch] gh/dskhudia/27/orig -> origin/gh/dskhudia/27/orig 2022-03-17T19:04:58.3652943Z * [new branch] gh/dzdang/1/base -> origin/gh/dzdang/1/base 2022-03-17T19:04:58.3653692Z * [new branch] gh/dzdang/1/orig -> origin/gh/dzdang/1/orig 2022-03-17T19:04:58.3654421Z * [new branch] gh/dzdang/10/base -> origin/gh/dzdang/10/base 2022-03-17T19:04:58.3655540Z * [new branch] gh/dzdang/10/orig -> origin/gh/dzdang/10/orig 2022-03-17T19:04:58.3657117Z * [new branch] gh/dzdang/11/base -> origin/gh/dzdang/11/base 2022-03-17T19:04:58.3658533Z * [new branch] gh/dzdang/11/orig -> origin/gh/dzdang/11/orig 2022-03-17T19:04:58.3660572Z * [new branch] gh/dzdang/13/base -> origin/gh/dzdang/13/base 2022-03-17T19:04:58.3662650Z * [new branch] gh/dzdang/13/head -> origin/gh/dzdang/13/head 2022-03-17T19:04:58.3663395Z * [new branch] gh/dzdang/13/orig -> origin/gh/dzdang/13/orig 2022-03-17T19:04:58.3664657Z * [new branch] gh/dzdang/14/base -> origin/gh/dzdang/14/base 2022-03-17T19:04:58.3666047Z * [new branch] gh/dzdang/14/head -> origin/gh/dzdang/14/head 2022-03-17T19:04:58.3667274Z * [new branch] gh/dzdang/14/orig -> origin/gh/dzdang/14/orig 2022-03-17T19:04:58.3669280Z * [new branch] gh/dzdang/15/base -> origin/gh/dzdang/15/base 2022-03-17T19:04:58.3670332Z * [new branch] gh/dzdang/15/head -> origin/gh/dzdang/15/head 2022-03-17T19:04:58.3671734Z * [new branch] gh/dzdang/15/orig -> origin/gh/dzdang/15/orig 2022-03-17T19:04:58.3673697Z * [new branch] gh/dzdang/2/base -> origin/gh/dzdang/2/base 2022-03-17T19:04:58.3674706Z * [new branch] gh/dzdang/2/orig -> origin/gh/dzdang/2/orig 2022-03-17T19:04:58.3676656Z * [new branch] gh/dzdang/29/base -> origin/gh/dzdang/29/base 2022-03-17T19:04:58.3677828Z * [new branch] gh/dzdang/29/head -> origin/gh/dzdang/29/head 2022-03-17T19:04:58.3679262Z * [new branch] gh/dzdang/29/orig -> origin/gh/dzdang/29/orig 2022-03-17T19:04:58.3681277Z * [new branch] gh/dzdang/30/base -> origin/gh/dzdang/30/base 2022-03-17T19:04:58.3682510Z * [new branch] gh/dzdang/30/head -> origin/gh/dzdang/30/head 2022-03-17T19:04:58.3684188Z * [new branch] gh/dzdang/30/orig -> origin/gh/dzdang/30/orig 2022-03-17T19:04:58.3685790Z * [new branch] gh/dzdang/36/orig -> origin/gh/dzdang/36/orig 2022-03-17T19:04:58.3687774Z * [new branch] gh/dzdang/37/base -> origin/gh/dzdang/37/base 2022-03-17T19:04:58.3688889Z * [new branch] gh/dzdang/37/head -> origin/gh/dzdang/37/head 2022-03-17T19:04:58.3690123Z * [new branch] gh/dzdang/37/orig -> origin/gh/dzdang/37/orig 2022-03-17T19:04:58.3691561Z * [new branch] gh/dzdang/38/base -> origin/gh/dzdang/38/base 2022-03-17T19:04:58.3692962Z * [new branch] gh/dzdang/38/head -> origin/gh/dzdang/38/head 2022-03-17T19:04:58.3694224Z * [new branch] gh/dzdang/38/orig -> origin/gh/dzdang/38/orig 2022-03-17T19:04:58.3696165Z * [new branch] gh/dzdang/4/base -> origin/gh/dzdang/4/base 2022-03-17T19:04:58.3697357Z * [new branch] gh/dzdang/4/orig -> origin/gh/dzdang/4/orig 2022-03-17T19:04:58.3699673Z * [new branch] gh/dzdang/42/base -> origin/gh/dzdang/42/base 2022-03-17T19:04:58.3700796Z * [new branch] gh/dzdang/42/head -> origin/gh/dzdang/42/head 2022-03-17T19:04:58.3702326Z * [new branch] gh/dzdang/42/orig -> origin/gh/dzdang/42/orig 2022-03-17T19:04:58.3704515Z * [new branch] gh/dzdang/47/base -> origin/gh/dzdang/47/base 2022-03-17T19:04:58.3706189Z * [new branch] gh/dzdang/47/head -> origin/gh/dzdang/47/head 2022-03-17T19:04:58.3707813Z * [new branch] gh/dzdang/47/orig -> origin/gh/dzdang/47/orig 2022-03-17T19:04:58.3709737Z * [new branch] gh/dzdang/48/base -> origin/gh/dzdang/48/base 2022-03-17T19:04:58.3710961Z * [new branch] gh/dzdang/48/head -> origin/gh/dzdang/48/head 2022-03-17T19:04:58.3712326Z * [new branch] gh/dzdang/48/orig -> origin/gh/dzdang/48/orig 2022-03-17T19:04:58.3714155Z * [new branch] gh/dzdang/49/base -> origin/gh/dzdang/49/base 2022-03-17T19:04:58.3715460Z * [new branch] gh/dzdang/49/head -> origin/gh/dzdang/49/head 2022-03-17T19:04:58.3716730Z * [new branch] gh/dzdang/49/orig -> origin/gh/dzdang/49/orig 2022-03-17T19:04:58.3718380Z * [new branch] gh/dzdang/5/base -> origin/gh/dzdang/5/base 2022-03-17T19:04:58.3719866Z * [new branch] gh/dzdang/5/orig -> origin/gh/dzdang/5/orig 2022-03-17T19:04:58.3722166Z * [new branch] gh/dzdang/50/base -> origin/gh/dzdang/50/base 2022-03-17T19:04:58.3723191Z * [new branch] gh/dzdang/50/head -> origin/gh/dzdang/50/head 2022-03-17T19:04:58.3724729Z * [new branch] gh/dzdang/50/orig -> origin/gh/dzdang/50/orig 2022-03-17T19:04:58.3726406Z * [new branch] gh/dzdang/51/base -> origin/gh/dzdang/51/base 2022-03-17T19:04:58.3727772Z * [new branch] gh/dzdang/51/head -> origin/gh/dzdang/51/head 2022-03-17T19:04:58.3729094Z * [new branch] gh/dzdang/51/orig -> origin/gh/dzdang/51/orig 2022-03-17T19:04:58.3731117Z * [new branch] gh/dzdang/52/base -> origin/gh/dzdang/52/base 2022-03-17T19:04:58.3732019Z * [new branch] gh/dzdang/52/head -> origin/gh/dzdang/52/head 2022-03-17T19:04:58.3732912Z * [new branch] gh/dzdang/52/orig -> origin/gh/dzdang/52/orig 2022-03-17T19:04:58.3735411Z * [new branch] gh/dzdang/53/base -> origin/gh/dzdang/53/base 2022-03-17T19:04:58.3736199Z * [new branch] gh/dzdang/53/head -> origin/gh/dzdang/53/head 2022-03-17T19:04:58.3737707Z * [new branch] gh/dzdang/53/orig -> origin/gh/dzdang/53/orig 2022-03-17T19:04:58.3739370Z * [new branch] gh/dzdang/54/base -> origin/gh/dzdang/54/base 2022-03-17T19:04:58.3740772Z * [new branch] gh/dzdang/54/head -> origin/gh/dzdang/54/head 2022-03-17T19:04:58.3741970Z * [new branch] gh/dzdang/54/orig -> origin/gh/dzdang/54/orig 2022-03-17T19:04:58.3743546Z * [new branch] gh/dzdang/55/base -> origin/gh/dzdang/55/base 2022-03-17T19:04:58.3744871Z * [new branch] gh/dzdang/55/head -> origin/gh/dzdang/55/head 2022-03-17T19:04:58.3746233Z * [new branch] gh/dzdang/55/orig -> origin/gh/dzdang/55/orig 2022-03-17T19:04:58.3747992Z * [new branch] gh/dzdang/56/base -> origin/gh/dzdang/56/base 2022-03-17T19:04:58.3749349Z * [new branch] gh/dzdang/56/head -> origin/gh/dzdang/56/head 2022-03-17T19:04:58.3750689Z * [new branch] gh/dzdang/56/orig -> origin/gh/dzdang/56/orig 2022-03-17T19:04:58.3752389Z * [new branch] gh/dzdang/57/base -> origin/gh/dzdang/57/base 2022-03-17T19:04:58.3754093Z * [new branch] gh/dzdang/57/head -> origin/gh/dzdang/57/head 2022-03-17T19:04:58.3755214Z * [new branch] gh/dzdang/57/orig -> origin/gh/dzdang/57/orig 2022-03-17T19:04:58.3756920Z * [new branch] gh/dzdang/58/base -> origin/gh/dzdang/58/base 2022-03-17T19:04:58.3758260Z * [new branch] gh/dzdang/58/head -> origin/gh/dzdang/58/head 2022-03-17T19:04:58.3759663Z * [new branch] gh/dzdang/58/orig -> origin/gh/dzdang/58/orig 2022-03-17T19:04:58.3761564Z * [new branch] gh/dzdang/59/base -> origin/gh/dzdang/59/base 2022-03-17T19:04:58.3763584Z * [new branch] gh/dzdang/59/head -> origin/gh/dzdang/59/head 2022-03-17T19:04:58.3764931Z * [new branch] gh/dzdang/59/orig -> origin/gh/dzdang/59/orig 2022-03-17T19:04:58.3766873Z * [new branch] gh/dzdang/6/base -> origin/gh/dzdang/6/base 2022-03-17T19:04:58.3768205Z * [new branch] gh/dzdang/6/orig -> origin/gh/dzdang/6/orig 2022-03-17T19:04:58.3769903Z * [new branch] gh/dzdang/7/base -> origin/gh/dzdang/7/base 2022-03-17T19:04:58.3771003Z * [new branch] gh/dzdang/7/orig -> origin/gh/dzdang/7/orig 2022-03-17T19:04:58.3772704Z * [new branch] gh/dzdang/8/base -> origin/gh/dzdang/8/base 2022-03-17T19:04:58.3774089Z * [new branch] gh/dzdang/8/orig -> origin/gh/dzdang/8/orig 2022-03-17T19:04:58.3775885Z * [new branch] gh/dzdang/9/base -> origin/gh/dzdang/9/base 2022-03-17T19:04:58.3777246Z * [new branch] gh/dzdang/9/orig -> origin/gh/dzdang/9/orig 2022-03-17T19:04:58.3779236Z * [new branch] gh/edmundw314/15/base -> origin/gh/edmundw314/15/base 2022-03-17T19:04:58.3780619Z * [new branch] gh/edmundw314/15/head -> origin/gh/edmundw314/15/head 2022-03-17T19:04:58.3782048Z * [new branch] gh/edmundw314/15/orig -> origin/gh/edmundw314/15/orig 2022-03-17T19:04:58.3783755Z * [new branch] gh/edmundw314/17/base -> origin/gh/edmundw314/17/base 2022-03-17T19:04:58.3785159Z * [new branch] gh/edmundw314/17/head -> origin/gh/edmundw314/17/head 2022-03-17T19:04:58.3786527Z * [new branch] gh/edmundw314/17/orig -> origin/gh/edmundw314/17/orig 2022-03-17T19:04:58.3788242Z * [new branch] gh/edmundw314/4/base -> origin/gh/edmundw314/4/base 2022-03-17T19:04:58.3789663Z * [new branch] gh/edmundw314/4/head -> origin/gh/edmundw314/4/head 2022-03-17T19:04:58.3790988Z * [new branch] gh/edmundw314/4/orig -> origin/gh/edmundw314/4/orig 2022-03-17T19:04:58.3792599Z * [new branch] gh/edmundw314/7/base -> origin/gh/edmundw314/7/base 2022-03-17T19:04:58.3793984Z * [new branch] gh/edmundw314/7/head -> origin/gh/edmundw314/7/head 2022-03-17T19:04:58.3795435Z * [new branch] gh/edmundw314/7/orig -> origin/gh/edmundw314/7/orig 2022-03-17T19:04:58.3797451Z * [new branch] gh/eellison/110/base -> origin/gh/eellison/110/base 2022-03-17T19:04:58.3798868Z * [new branch] gh/eellison/110/head -> origin/gh/eellison/110/head 2022-03-17T19:04:58.3800195Z * [new branch] gh/eellison/110/orig -> origin/gh/eellison/110/orig 2022-03-17T19:04:58.3802019Z * [new branch] gh/eellison/111/base -> origin/gh/eellison/111/base 2022-03-17T19:04:58.3803661Z * [new branch] gh/eellison/111/head -> origin/gh/eellison/111/head 2022-03-17T19:04:58.3804985Z * [new branch] gh/eellison/111/orig -> origin/gh/eellison/111/orig 2022-03-17T19:04:58.3806629Z * [new branch] gh/eellison/117/base -> origin/gh/eellison/117/base 2022-03-17T19:04:58.3807934Z * [new branch] gh/eellison/117/head -> origin/gh/eellison/117/head 2022-03-17T19:04:58.3809435Z * [new branch] gh/eellison/117/orig -> origin/gh/eellison/117/orig 2022-03-17T19:04:58.3810697Z * [new branch] gh/eellison/118/base -> origin/gh/eellison/118/base 2022-03-17T19:04:58.3812053Z * [new branch] gh/eellison/118/head -> origin/gh/eellison/118/head 2022-03-17T19:04:58.3813251Z * [new branch] gh/eellison/118/orig -> origin/gh/eellison/118/orig 2022-03-17T19:04:58.3815502Z * [new branch] gh/eellison/119/base -> origin/gh/eellison/119/base 2022-03-17T19:04:58.3816676Z * [new branch] gh/eellison/119/head -> origin/gh/eellison/119/head 2022-03-17T19:04:58.3817915Z * [new branch] gh/eellison/119/orig -> origin/gh/eellison/119/orig 2022-03-17T19:04:58.3819875Z * [new branch] gh/eellison/129/base -> origin/gh/eellison/129/base 2022-03-17T19:04:58.3821020Z * [new branch] gh/eellison/129/head -> origin/gh/eellison/129/head 2022-03-17T19:04:58.3822441Z * [new branch] gh/eellison/129/orig -> origin/gh/eellison/129/orig 2022-03-17T19:04:58.3824157Z * [new branch] gh/eellison/139/base -> origin/gh/eellison/139/base 2022-03-17T19:04:58.3825342Z * [new branch] gh/eellison/139/head -> origin/gh/eellison/139/head 2022-03-17T19:04:58.3826733Z * [new branch] gh/eellison/139/orig -> origin/gh/eellison/139/orig 2022-03-17T19:04:58.3828529Z * [new branch] gh/eellison/142/base -> origin/gh/eellison/142/base 2022-03-17T19:04:58.3829986Z * [new branch] gh/eellison/142/head -> origin/gh/eellison/142/head 2022-03-17T19:04:58.3831238Z * [new branch] gh/eellison/142/orig -> origin/gh/eellison/142/orig 2022-03-17T19:04:58.3833156Z * [new branch] gh/eellison/154/base -> origin/gh/eellison/154/base 2022-03-17T19:04:58.3834335Z * [new branch] gh/eellison/154/head -> origin/gh/eellison/154/head 2022-03-17T19:04:58.3835810Z * [new branch] gh/eellison/154/orig -> origin/gh/eellison/154/orig 2022-03-17T19:04:58.3837495Z * [new branch] gh/eellison/169/base -> origin/gh/eellison/169/base 2022-03-17T19:04:58.3838795Z * [new branch] gh/eellison/169/head -> origin/gh/eellison/169/head 2022-03-17T19:04:58.3839835Z * [new branch] gh/eellison/169/orig -> origin/gh/eellison/169/orig 2022-03-17T19:04:58.3841601Z * [new branch] gh/eellison/179/base -> origin/gh/eellison/179/base 2022-03-17T19:04:58.3843096Z * [new branch] gh/eellison/179/head -> origin/gh/eellison/179/head 2022-03-17T19:04:58.3844310Z * [new branch] gh/eellison/179/orig -> origin/gh/eellison/179/orig 2022-03-17T19:04:58.3846032Z * [new branch] gh/eellison/182/base -> origin/gh/eellison/182/base 2022-03-17T19:04:58.3847511Z * [new branch] gh/eellison/182/head -> origin/gh/eellison/182/head 2022-03-17T19:04:58.3848735Z * [new branch] gh/eellison/182/orig -> origin/gh/eellison/182/orig 2022-03-17T19:04:58.3850610Z * [new branch] gh/eellison/183/base -> origin/gh/eellison/183/base 2022-03-17T19:04:58.3851952Z * [new branch] gh/eellison/183/head -> origin/gh/eellison/183/head 2022-03-17T19:04:58.3853327Z * [new branch] gh/eellison/183/orig -> origin/gh/eellison/183/orig 2022-03-17T19:04:58.3854989Z * [new branch] gh/eellison/191/base -> origin/gh/eellison/191/base 2022-03-17T19:04:58.3856383Z * [new branch] gh/eellison/191/head -> origin/gh/eellison/191/head 2022-03-17T19:04:58.3857756Z * [new branch] gh/eellison/191/orig -> origin/gh/eellison/191/orig 2022-03-17T19:04:58.3859421Z * [new branch] gh/eellison/193/base -> origin/gh/eellison/193/base 2022-03-17T19:04:58.3860778Z * [new branch] gh/eellison/193/head -> origin/gh/eellison/193/head 2022-03-17T19:04:58.3862192Z * [new branch] gh/eellison/193/orig -> origin/gh/eellison/193/orig 2022-03-17T19:04:58.3863935Z * [new branch] gh/eellison/195/base -> origin/gh/eellison/195/base 2022-03-17T19:04:58.3865171Z * [new branch] gh/eellison/195/head -> origin/gh/eellison/195/head 2022-03-17T19:04:58.3866473Z * [new branch] gh/eellison/195/orig -> origin/gh/eellison/195/orig 2022-03-17T19:04:58.3867990Z * [new branch] gh/eellison/196/base -> origin/gh/eellison/196/base 2022-03-17T19:04:58.3873634Z * [new branch] gh/eellison/196/head -> origin/gh/eellison/196/head 2022-03-17T19:04:58.3874043Z * [new branch] gh/eellison/196/orig -> origin/gh/eellison/196/orig 2022-03-17T19:04:58.3874460Z * [new branch] gh/eellison/209/base -> origin/gh/eellison/209/base 2022-03-17T19:04:58.3874855Z * [new branch] gh/eellison/209/head -> origin/gh/eellison/209/head 2022-03-17T19:04:58.3875408Z * [new branch] gh/eellison/209/orig -> origin/gh/eellison/209/orig 2022-03-17T19:04:58.3876913Z * [new branch] gh/eellison/232/base -> origin/gh/eellison/232/base 2022-03-17T19:04:58.3878201Z * [new branch] gh/eellison/232/head -> origin/gh/eellison/232/head 2022-03-17T19:04:58.3879556Z * [new branch] gh/eellison/232/orig -> origin/gh/eellison/232/orig 2022-03-17T19:04:58.3881221Z * [new branch] gh/eellison/234/base -> origin/gh/eellison/234/base 2022-03-17T19:04:58.3882785Z * [new branch] gh/eellison/234/head -> origin/gh/eellison/234/head 2022-03-17T19:04:58.3884304Z * [new branch] gh/eellison/234/orig -> origin/gh/eellison/234/orig 2022-03-17T19:04:58.3885971Z * [new branch] gh/eellison/238/base -> origin/gh/eellison/238/base 2022-03-17T19:04:58.3887323Z * [new branch] gh/eellison/238/head -> origin/gh/eellison/238/head 2022-03-17T19:04:58.3888682Z * [new branch] gh/eellison/238/orig -> origin/gh/eellison/238/orig 2022-03-17T19:04:58.3890441Z * [new branch] gh/eellison/262/base -> origin/gh/eellison/262/base 2022-03-17T19:04:58.3891597Z * [new branch] gh/eellison/262/head -> origin/gh/eellison/262/head 2022-03-17T19:04:58.3892906Z * [new branch] gh/eellison/262/orig -> origin/gh/eellison/262/orig 2022-03-17T19:04:58.3894355Z * [new branch] gh/eellison/263/base -> origin/gh/eellison/263/base 2022-03-17T19:04:58.3895690Z * [new branch] gh/eellison/263/head -> origin/gh/eellison/263/head 2022-03-17T19:04:58.3896993Z * [new branch] gh/eellison/263/orig -> origin/gh/eellison/263/orig 2022-03-17T19:04:58.3898484Z * [new branch] gh/eellison/264/base -> origin/gh/eellison/264/base 2022-03-17T19:04:58.3899811Z * [new branch] gh/eellison/264/head -> origin/gh/eellison/264/head 2022-03-17T19:04:58.3901112Z * [new branch] gh/eellison/264/orig -> origin/gh/eellison/264/orig 2022-03-17T19:04:58.3902679Z * [new branch] gh/eellison/265/base -> origin/gh/eellison/265/base 2022-03-17T19:04:58.3903941Z * [new branch] gh/eellison/265/head -> origin/gh/eellison/265/head 2022-03-17T19:04:58.3905241Z * [new branch] gh/eellison/265/orig -> origin/gh/eellison/265/orig 2022-03-17T19:04:58.3906843Z * [new branch] gh/eellison/272/base -> origin/gh/eellison/272/base 2022-03-17T19:04:58.3908179Z * [new branch] gh/eellison/272/head -> origin/gh/eellison/272/head 2022-03-17T19:04:58.3909579Z * [new branch] gh/eellison/272/orig -> origin/gh/eellison/272/orig 2022-03-17T19:04:58.3911261Z * [new branch] gh/eellison/274/base -> origin/gh/eellison/274/base 2022-03-17T19:04:58.3912742Z * [new branch] gh/eellison/274/head -> origin/gh/eellison/274/head 2022-03-17T19:04:58.3913954Z * [new branch] gh/eellison/274/orig -> origin/gh/eellison/274/orig 2022-03-17T19:04:58.3915830Z * [new branch] gh/eellison/275/base -> origin/gh/eellison/275/base 2022-03-17T19:04:58.3917099Z * [new branch] gh/eellison/275/head -> origin/gh/eellison/275/head 2022-03-17T19:04:58.3918403Z * [new branch] gh/eellison/275/orig -> origin/gh/eellison/275/orig 2022-03-17T19:04:58.3920099Z * [new branch] gh/eellison/276/base -> origin/gh/eellison/276/base 2022-03-17T19:04:58.3921392Z * [new branch] gh/eellison/276/head -> origin/gh/eellison/276/head 2022-03-17T19:04:58.3925153Z * [new branch] gh/eellison/276/orig -> origin/gh/eellison/276/orig 2022-03-17T19:04:58.3926685Z * [new branch] gh/eellison/278/base -> origin/gh/eellison/278/base 2022-03-17T19:04:58.3928494Z * [new branch] gh/eellison/278/head -> origin/gh/eellison/278/head 2022-03-17T19:04:58.3930098Z * [new branch] gh/eellison/278/orig -> origin/gh/eellison/278/orig 2022-03-17T19:04:58.3931974Z * [new branch] gh/eellison/279/base -> origin/gh/eellison/279/base 2022-03-17T19:04:58.3932889Z * [new branch] gh/eellison/279/head -> origin/gh/eellison/279/head 2022-03-17T19:04:58.3935183Z * [new branch] gh/eellison/279/orig -> origin/gh/eellison/279/orig 2022-03-17T19:04:58.3936440Z * [new branch] gh/eellison/280/base -> origin/gh/eellison/280/base 2022-03-17T19:04:58.3937907Z * [new branch] gh/eellison/280/head -> origin/gh/eellison/280/head 2022-03-17T19:04:58.3939949Z * [new branch] gh/eellison/281/base -> origin/gh/eellison/281/base 2022-03-17T19:04:58.3941448Z * [new branch] gh/eellison/281/head -> origin/gh/eellison/281/head 2022-03-17T19:04:58.3942932Z * [new branch] gh/eellison/281/orig -> origin/gh/eellison/281/orig 2022-03-17T19:04:58.3944909Z * [new branch] gh/eellison/282/base -> origin/gh/eellison/282/base 2022-03-17T19:04:58.3946578Z * [new branch] gh/eellison/282/head -> origin/gh/eellison/282/head 2022-03-17T19:04:58.3947895Z * [new branch] gh/eellison/282/orig -> origin/gh/eellison/282/orig 2022-03-17T19:04:58.3949853Z * [new branch] gh/eellison/283/base -> origin/gh/eellison/283/base 2022-03-17T19:04:58.3951237Z * [new branch] gh/eellison/283/head -> origin/gh/eellison/283/head 2022-03-17T19:04:58.3952732Z * [new branch] gh/eellison/283/orig -> origin/gh/eellison/283/orig 2022-03-17T19:04:58.3954577Z * [new branch] gh/eellison/284/base -> origin/gh/eellison/284/base 2022-03-17T19:04:58.3955993Z * [new branch] gh/eellison/284/head -> origin/gh/eellison/284/head 2022-03-17T19:04:58.3957457Z * [new branch] gh/eellison/284/orig -> origin/gh/eellison/284/orig 2022-03-17T19:04:58.3959375Z * [new branch] gh/eellison/285/base -> origin/gh/eellison/285/base 2022-03-17T19:04:58.3960814Z * [new branch] gh/eellison/285/head -> origin/gh/eellison/285/head 2022-03-17T19:04:58.3962351Z * [new branch] gh/eellison/285/orig -> origin/gh/eellison/285/orig 2022-03-17T19:04:58.3964578Z * [new branch] gh/eellison/286/base -> origin/gh/eellison/286/base 2022-03-17T19:04:58.3965860Z * [new branch] gh/eellison/286/head -> origin/gh/eellison/286/head 2022-03-17T19:04:58.3967429Z * [new branch] gh/eellison/286/orig -> origin/gh/eellison/286/orig 2022-03-17T19:04:58.3969330Z * [new branch] gh/eellison/49/base -> origin/gh/eellison/49/base 2022-03-17T19:04:58.3971312Z * [new branch] gh/eellison/49/head -> origin/gh/eellison/49/head 2022-03-17T19:04:58.3974257Z * [new branch] gh/eellison/49/orig -> origin/gh/eellison/49/orig 2022-03-17T19:04:58.3978104Z * [new branch] gh/ejguan/24/base -> origin/gh/ejguan/24/base 2022-03-17T19:04:58.3980941Z * [new branch] gh/ejguan/24/head -> origin/gh/ejguan/24/head 2022-03-17T19:04:58.3983602Z * [new branch] gh/ejguan/24/orig -> origin/gh/ejguan/24/orig 2022-03-17T19:04:58.3986778Z * [new branch] gh/ejguan/25/base -> origin/gh/ejguan/25/base 2022-03-17T19:04:58.3989461Z * [new branch] gh/ejguan/25/head -> origin/gh/ejguan/25/head 2022-03-17T19:04:58.3992161Z * [new branch] gh/ejguan/25/orig -> origin/gh/ejguan/25/orig 2022-03-17T19:04:58.3995331Z * [new branch] gh/ejguan/26/base -> origin/gh/ejguan/26/base 2022-03-17T19:04:58.3998100Z * [new branch] gh/ejguan/26/head -> origin/gh/ejguan/26/head 2022-03-17T19:04:58.4000415Z * [new branch] gh/ejguan/26/orig -> origin/gh/ejguan/26/orig 2022-03-17T19:04:58.4004433Z * [new branch] gh/ejguan/27/base -> origin/gh/ejguan/27/base 2022-03-17T19:04:58.4007723Z * [new branch] gh/ejguan/27/head -> origin/gh/ejguan/27/head 2022-03-17T19:04:58.4010525Z * [new branch] gh/ejguan/27/orig -> origin/gh/ejguan/27/orig 2022-03-17T19:04:58.4013932Z * [new branch] gh/ejguan/3/base -> origin/gh/ejguan/3/base 2022-03-17T19:04:58.4016794Z * [new branch] gh/ejguan/3/head -> origin/gh/ejguan/3/head 2022-03-17T19:04:58.4019351Z * [new branch] gh/ejguan/3/orig -> origin/gh/ejguan/3/orig 2022-03-17T19:04:58.4022861Z * [new branch] gh/ejguan/30/base -> origin/gh/ejguan/30/base 2022-03-17T19:04:58.4025597Z * [new branch] gh/ejguan/30/head -> origin/gh/ejguan/30/head 2022-03-17T19:04:58.4029027Z * [new branch] gh/ejguan/33/base -> origin/gh/ejguan/33/base 2022-03-17T19:04:58.4031703Z * [new branch] gh/ejguan/33/head -> origin/gh/ejguan/33/head 2022-03-17T19:04:58.4035242Z * [new branch] gh/ejguan/40/base -> origin/gh/ejguan/40/base 2022-03-17T19:04:58.4038362Z * [new branch] gh/ejguan/40/head -> origin/gh/ejguan/40/head 2022-03-17T19:04:58.4041257Z * [new branch] gh/ejguan/40/orig -> origin/gh/ejguan/40/orig 2022-03-17T19:04:58.4045313Z * [new branch] gh/ejguan/68/base -> origin/gh/ejguan/68/base 2022-03-17T19:04:58.4048086Z * [new branch] gh/ejguan/68/head -> origin/gh/ejguan/68/head 2022-03-17T19:04:58.4050792Z * [new branch] gh/ejguan/68/orig -> origin/gh/ejguan/68/orig 2022-03-17T19:04:58.4054468Z * [new branch] gh/ejguan/71/base -> origin/gh/ejguan/71/base 2022-03-17T19:04:58.4057168Z * [new branch] gh/ejguan/71/head -> origin/gh/ejguan/71/head 2022-03-17T19:04:58.4059891Z * [new branch] gh/ejguan/71/orig -> origin/gh/ejguan/71/orig 2022-03-17T19:04:58.4062919Z * [new branch] gh/ejguan/72/base -> origin/gh/ejguan/72/base 2022-03-17T19:04:58.4065774Z * [new branch] gh/ejguan/72/head -> origin/gh/ejguan/72/head 2022-03-17T19:04:58.4068654Z * [new branch] gh/ejguan/72/orig -> origin/gh/ejguan/72/orig 2022-03-17T19:04:58.4072345Z * [new branch] gh/ejguan/73/base -> origin/gh/ejguan/73/base 2022-03-17T19:04:58.4074906Z * [new branch] gh/ejguan/73/head -> origin/gh/ejguan/73/head 2022-03-17T19:04:58.4077785Z * [new branch] gh/ejguan/73/orig -> origin/gh/ejguan/73/orig 2022-03-17T19:04:58.4081352Z * [new branch] gh/ejguan/74/base -> origin/gh/ejguan/74/base 2022-03-17T19:04:58.4084561Z * [new branch] gh/ejguan/74/head -> origin/gh/ejguan/74/head 2022-03-17T19:04:58.4087322Z * [new branch] gh/ejguan/74/orig -> origin/gh/ejguan/74/orig 2022-03-17T19:04:58.4091287Z * [new branch] gh/ejguan/89/base -> origin/gh/ejguan/89/base 2022-03-17T19:04:58.4093941Z * [new branch] gh/ejguan/89/head -> origin/gh/ejguan/89/head 2022-03-17T19:04:58.4096654Z * [new branch] gh/ejguan/89/orig -> origin/gh/ejguan/89/orig 2022-03-17T19:04:58.4100684Z * [new branch] gh/erjia-guan/1/base -> origin/gh/erjia-guan/1/base 2022-03-17T19:04:58.4103310Z * [new branch] gh/erjia-guan/1/head -> origin/gh/erjia-guan/1/head 2022-03-17T19:04:58.4106048Z * [new branch] gh/erjia-guan/1/orig -> origin/gh/erjia-guan/1/orig 2022-03-17T19:04:58.4109998Z * [new branch] gh/esqu1/1/base -> origin/gh/esqu1/1/base 2022-03-17T19:04:58.4112698Z * [new branch] gh/esqu1/1/head -> origin/gh/esqu1/1/head 2022-03-17T19:04:58.4115478Z * [new branch] gh/esqu1/1/orig -> origin/gh/esqu1/1/orig 2022-03-17T19:04:58.4119424Z * [new branch] gh/ethch18/1/base -> origin/gh/ethch18/1/base 2022-03-17T19:04:58.4123717Z * [new branch] gh/ezyang/1009/base -> origin/gh/ezyang/1009/base 2022-03-17T19:04:58.4126391Z * [new branch] gh/ezyang/1009/head -> origin/gh/ezyang/1009/head 2022-03-17T19:04:58.4129137Z * [new branch] gh/ezyang/1009/orig -> origin/gh/ezyang/1009/orig 2022-03-17T19:04:58.4132046Z * [new branch] gh/ezyang/1035/base -> origin/gh/ezyang/1035/base 2022-03-17T19:04:58.4133294Z * [new branch] gh/ezyang/1035/head -> origin/gh/ezyang/1035/head 2022-03-17T19:04:58.4134828Z * [new branch] gh/ezyang/1035/orig -> origin/gh/ezyang/1035/orig 2022-03-17T19:04:58.4136763Z * [new branch] gh/ezyang/1053/base -> origin/gh/ezyang/1053/base 2022-03-17T19:04:58.4138302Z * [new branch] gh/ezyang/1053/head -> origin/gh/ezyang/1053/head 2022-03-17T19:04:58.4139935Z * [new branch] gh/ezyang/1053/orig -> origin/gh/ezyang/1053/orig 2022-03-17T19:04:58.4141826Z * [new branch] gh/ezyang/1054/base -> origin/gh/ezyang/1054/base 2022-03-17T19:04:58.4143154Z * [new branch] gh/ezyang/1054/head -> origin/gh/ezyang/1054/head 2022-03-17T19:04:58.4144613Z * [new branch] gh/ezyang/1054/orig -> origin/gh/ezyang/1054/orig 2022-03-17T19:04:58.4146593Z * [new branch] gh/ezyang/1068/base -> origin/gh/ezyang/1068/base 2022-03-17T19:04:58.4148031Z * [new branch] gh/ezyang/1068/head -> origin/gh/ezyang/1068/head 2022-03-17T19:04:58.4149464Z * [new branch] gh/ezyang/1068/orig -> origin/gh/ezyang/1068/orig 2022-03-17T19:04:58.4151328Z * [new branch] gh/ezyang/1070/base -> origin/gh/ezyang/1070/base 2022-03-17T19:04:58.4152723Z * [new branch] gh/ezyang/1070/head -> origin/gh/ezyang/1070/head 2022-03-17T19:04:58.4154187Z * [new branch] gh/ezyang/1070/orig -> origin/gh/ezyang/1070/orig 2022-03-17T19:04:58.4156144Z * [new branch] gh/ezyang/1071/base -> origin/gh/ezyang/1071/base 2022-03-17T19:04:58.4157538Z * [new branch] gh/ezyang/1071/head -> origin/gh/ezyang/1071/head 2022-03-17T19:04:58.4159054Z * [new branch] gh/ezyang/1071/orig -> origin/gh/ezyang/1071/orig 2022-03-17T19:04:58.4160696Z * [new branch] gh/ezyang/1073/base -> origin/gh/ezyang/1073/base 2022-03-17T19:04:58.4162182Z * [new branch] gh/ezyang/1073/head -> origin/gh/ezyang/1073/head 2022-03-17T19:04:58.4163865Z * [new branch] gh/ezyang/1073/orig -> origin/gh/ezyang/1073/orig 2022-03-17T19:04:58.4165736Z * [new branch] gh/ezyang/1075/base -> origin/gh/ezyang/1075/base 2022-03-17T19:04:58.4167431Z * [new branch] gh/ezyang/1075/head -> origin/gh/ezyang/1075/head 2022-03-17T19:04:58.4168713Z * [new branch] gh/ezyang/1075/orig -> origin/gh/ezyang/1075/orig 2022-03-17T19:04:58.4170653Z * [new branch] gh/ezyang/1081/base -> origin/gh/ezyang/1081/base 2022-03-17T19:04:58.4172187Z * [new branch] gh/ezyang/1081/head -> origin/gh/ezyang/1081/head 2022-03-17T19:04:58.4173752Z * [new branch] gh/ezyang/1081/orig -> origin/gh/ezyang/1081/orig 2022-03-17T19:04:58.4175585Z * [new branch] gh/ezyang/1086/base -> origin/gh/ezyang/1086/base 2022-03-17T19:04:58.4177081Z * [new branch] gh/ezyang/1086/head -> origin/gh/ezyang/1086/head 2022-03-17T19:04:58.4178530Z * [new branch] gh/ezyang/1086/orig -> origin/gh/ezyang/1086/orig 2022-03-17T19:04:58.4180578Z * [new branch] gh/ezyang/1090/base -> origin/gh/ezyang/1090/base 2022-03-17T19:04:58.4182257Z * [new branch] gh/ezyang/1090/head -> origin/gh/ezyang/1090/head 2022-03-17T19:04:58.4183968Z * [new branch] gh/ezyang/1090/orig -> origin/gh/ezyang/1090/orig 2022-03-17T19:04:58.4185932Z * [new branch] gh/ezyang/1093/base -> origin/gh/ezyang/1093/base 2022-03-17T19:04:58.4187546Z * [new branch] gh/ezyang/1093/head -> origin/gh/ezyang/1093/head 2022-03-17T19:04:58.4189098Z * [new branch] gh/ezyang/1093/orig -> origin/gh/ezyang/1093/orig 2022-03-17T19:04:58.4190941Z * [new branch] gh/ezyang/1096/base -> origin/gh/ezyang/1096/base 2022-03-17T19:04:58.4192573Z * [new branch] gh/ezyang/1096/head -> origin/gh/ezyang/1096/head 2022-03-17T19:04:58.4194074Z * [new branch] gh/ezyang/1096/orig -> origin/gh/ezyang/1096/orig 2022-03-17T19:04:58.4195929Z * [new branch] gh/ezyang/1097/base -> origin/gh/ezyang/1097/base 2022-03-17T19:04:58.4197426Z * [new branch] gh/ezyang/1097/head -> origin/gh/ezyang/1097/head 2022-03-17T19:04:58.4198946Z * [new branch] gh/ezyang/1097/orig -> origin/gh/ezyang/1097/orig 2022-03-17T19:04:58.4200791Z * [new branch] gh/ezyang/1098/base -> origin/gh/ezyang/1098/base 2022-03-17T19:04:58.4202349Z * [new branch] gh/ezyang/1098/head -> origin/gh/ezyang/1098/head 2022-03-17T19:04:58.4203996Z * [new branch] gh/ezyang/1098/orig -> origin/gh/ezyang/1098/orig 2022-03-17T19:04:58.4205799Z * [new branch] gh/ezyang/1099/base -> origin/gh/ezyang/1099/base 2022-03-17T19:04:58.4207270Z * [new branch] gh/ezyang/1099/head -> origin/gh/ezyang/1099/head 2022-03-17T19:04:58.4208690Z * [new branch] gh/ezyang/1099/orig -> origin/gh/ezyang/1099/orig 2022-03-17T19:04:58.4210364Z * [new branch] gh/ezyang/1100/base -> origin/gh/ezyang/1100/base 2022-03-17T19:04:58.4211638Z * [new branch] gh/ezyang/1100/head -> origin/gh/ezyang/1100/head 2022-03-17T19:04:58.4213803Z * [new branch] gh/ezyang/1100/orig -> origin/gh/ezyang/1100/orig 2022-03-17T19:04:58.4215587Z * [new branch] gh/ezyang/1101/base -> origin/gh/ezyang/1101/base 2022-03-17T19:04:58.4216772Z * [new branch] gh/ezyang/1101/head -> origin/gh/ezyang/1101/head 2022-03-17T19:04:58.4218344Z * [new branch] gh/ezyang/1101/orig -> origin/gh/ezyang/1101/orig 2022-03-17T19:04:58.4220311Z * [new branch] gh/ezyang/1102/base -> origin/gh/ezyang/1102/base 2022-03-17T19:04:58.4221840Z * [new branch] gh/ezyang/1102/head -> origin/gh/ezyang/1102/head 2022-03-17T19:04:58.4223305Z * [new branch] gh/ezyang/1102/orig -> origin/gh/ezyang/1102/orig 2022-03-17T19:04:58.4225182Z * [new branch] gh/ezyang/1103/base -> origin/gh/ezyang/1103/base 2022-03-17T19:04:58.4226617Z * [new branch] gh/ezyang/1103/head -> origin/gh/ezyang/1103/head 2022-03-17T19:04:58.4228112Z * [new branch] gh/ezyang/1103/orig -> origin/gh/ezyang/1103/orig 2022-03-17T19:04:58.4229806Z * [new branch] gh/ezyang/1104/base -> origin/gh/ezyang/1104/base 2022-03-17T19:04:58.4231373Z * [new branch] gh/ezyang/1104/head -> origin/gh/ezyang/1104/head 2022-03-17T19:04:58.4232812Z * [new branch] gh/ezyang/1104/orig -> origin/gh/ezyang/1104/orig 2022-03-17T19:04:58.4234828Z * [new branch] gh/ezyang/1105/base -> origin/gh/ezyang/1105/base 2022-03-17T19:04:58.4236200Z * [new branch] gh/ezyang/1105/head -> origin/gh/ezyang/1105/head 2022-03-17T19:04:58.4237905Z * [new branch] gh/ezyang/1106/base -> origin/gh/ezyang/1106/base 2022-03-17T19:04:58.4239268Z * [new branch] gh/ezyang/1106/head -> origin/gh/ezyang/1106/head 2022-03-17T19:04:58.4240576Z * [new branch] gh/ezyang/1106/orig -> origin/gh/ezyang/1106/orig 2022-03-17T19:04:58.4242526Z * [new branch] gh/ezyang/1107/base -> origin/gh/ezyang/1107/base 2022-03-17T19:04:58.4244510Z * [new branch] gh/ezyang/1107/head -> origin/gh/ezyang/1107/head 2022-03-17T19:04:58.4245882Z * [new branch] gh/ezyang/1107/orig -> origin/gh/ezyang/1107/orig 2022-03-17T19:04:58.4247725Z * [new branch] gh/ezyang/269/base -> origin/gh/ezyang/269/base 2022-03-17T19:04:58.4249228Z * [new branch] gh/ezyang/269/head -> origin/gh/ezyang/269/head 2022-03-17T19:04:58.4250693Z * [new branch] gh/ezyang/269/orig -> origin/gh/ezyang/269/orig 2022-03-17T19:04:58.4252565Z * [new branch] gh/ezyang/301/base -> origin/gh/ezyang/301/base 2022-03-17T19:04:58.4254113Z * [new branch] gh/ezyang/301/head -> origin/gh/ezyang/301/head 2022-03-17T19:04:58.4255563Z * [new branch] gh/ezyang/301/orig -> origin/gh/ezyang/301/orig 2022-03-17T19:04:58.4257396Z * [new branch] gh/ezyang/366/head -> origin/gh/ezyang/366/head 2022-03-17T19:04:58.4258867Z * [new branch] gh/ezyang/366/orig -> origin/gh/ezyang/366/orig 2022-03-17T19:04:58.4260598Z * [new branch] gh/ezyang/371/base -> origin/gh/ezyang/371/base 2022-03-17T19:04:58.4262835Z * [new branch] gh/ezyang/371/head -> origin/gh/ezyang/371/head 2022-03-17T19:04:58.4264877Z * [new branch] gh/ezyang/371/orig -> origin/gh/ezyang/371/orig 2022-03-17T19:04:58.4267033Z * [new branch] gh/ezyang/401/base -> origin/gh/ezyang/401/base 2022-03-17T19:04:58.4268484Z * [new branch] gh/ezyang/401/head -> origin/gh/ezyang/401/head 2022-03-17T19:04:58.4270163Z * [new branch] gh/ezyang/479/base -> origin/gh/ezyang/479/base 2022-03-17T19:04:58.4271532Z * [new branch] gh/ezyang/479/head -> origin/gh/ezyang/479/head 2022-03-17T19:04:58.4272910Z * [new branch] gh/ezyang/479/orig -> origin/gh/ezyang/479/orig 2022-03-17T19:04:58.4274666Z * [new branch] gh/ezyang/495/head -> origin/gh/ezyang/495/head 2022-03-17T19:04:58.4276033Z * [new branch] gh/ezyang/495/orig -> origin/gh/ezyang/495/orig 2022-03-17T19:04:58.4277852Z * [new branch] gh/ezyang/497/base -> origin/gh/ezyang/497/base 2022-03-17T19:04:58.4279386Z * [new branch] gh/ezyang/497/head -> origin/gh/ezyang/497/head 2022-03-17T19:04:58.4280709Z * [new branch] gh/ezyang/497/orig -> origin/gh/ezyang/497/orig 2022-03-17T19:04:58.4282263Z * [new branch] gh/ezyang/501/base -> origin/gh/ezyang/501/base 2022-03-17T19:04:58.4283884Z * [new branch] gh/ezyang/501/head -> origin/gh/ezyang/501/head 2022-03-17T19:04:58.4285214Z * [new branch] gh/ezyang/501/orig -> origin/gh/ezyang/501/orig 2022-03-17T19:04:58.4287192Z * [new branch] gh/ezyang/502/base -> origin/gh/ezyang/502/base 2022-03-17T19:04:58.4288531Z * [new branch] gh/ezyang/502/head -> origin/gh/ezyang/502/head 2022-03-17T19:04:58.4289888Z * [new branch] gh/ezyang/502/orig -> origin/gh/ezyang/502/orig 2022-03-17T19:04:58.4291425Z * [new branch] gh/ezyang/503/base -> origin/gh/ezyang/503/base 2022-03-17T19:04:58.4292778Z * [new branch] gh/ezyang/503/head -> origin/gh/ezyang/503/head 2022-03-17T19:04:58.4294134Z * [new branch] gh/ezyang/503/orig -> origin/gh/ezyang/503/orig 2022-03-17T19:04:58.4295646Z * [new branch] gh/ezyang/504/base -> origin/gh/ezyang/504/base 2022-03-17T19:04:58.4297065Z * [new branch] gh/ezyang/504/head -> origin/gh/ezyang/504/head 2022-03-17T19:04:58.4298479Z * [new branch] gh/ezyang/504/orig -> origin/gh/ezyang/504/orig 2022-03-17T19:04:58.4300037Z * [new branch] gh/ezyang/505/base -> origin/gh/ezyang/505/base 2022-03-17T19:04:58.4301376Z * [new branch] gh/ezyang/505/head -> origin/gh/ezyang/505/head 2022-03-17T19:04:58.4302713Z * [new branch] gh/ezyang/505/orig -> origin/gh/ezyang/505/orig 2022-03-17T19:04:58.4304178Z * [new branch] gh/ezyang/506/base -> origin/gh/ezyang/506/base 2022-03-17T19:04:58.4305823Z * [new branch] gh/ezyang/506/head -> origin/gh/ezyang/506/head 2022-03-17T19:04:58.4307350Z * [new branch] gh/ezyang/506/orig -> origin/gh/ezyang/506/orig 2022-03-17T19:04:58.4309059Z * [new branch] gh/ezyang/546/base -> origin/gh/ezyang/546/base 2022-03-17T19:04:58.4310487Z * [new branch] gh/ezyang/546/head -> origin/gh/ezyang/546/head 2022-03-17T19:04:58.4311578Z * [new branch] gh/ezyang/546/orig -> origin/gh/ezyang/546/orig 2022-03-17T19:04:58.4313183Z * [new branch] gh/ezyang/547/base -> origin/gh/ezyang/547/base 2022-03-17T19:04:58.4314483Z * [new branch] gh/ezyang/547/head -> origin/gh/ezyang/547/head 2022-03-17T19:04:58.4315791Z * [new branch] gh/ezyang/547/orig -> origin/gh/ezyang/547/orig 2022-03-17T19:04:58.4317289Z * [new branch] gh/ezyang/548/base -> origin/gh/ezyang/548/base 2022-03-17T19:04:58.4318691Z * [new branch] gh/ezyang/548/head -> origin/gh/ezyang/548/head 2022-03-17T19:04:58.4320065Z * [new branch] gh/ezyang/548/orig -> origin/gh/ezyang/548/orig 2022-03-17T19:04:58.4321584Z * [new branch] gh/ezyang/549/base -> origin/gh/ezyang/549/base 2022-03-17T19:04:58.4323202Z * [new branch] gh/ezyang/549/head -> origin/gh/ezyang/549/head 2022-03-17T19:04:58.4324547Z * [new branch] gh/ezyang/549/orig -> origin/gh/ezyang/549/orig 2022-03-17T19:04:58.4326009Z * [new branch] gh/ezyang/550/base -> origin/gh/ezyang/550/base 2022-03-17T19:04:58.4327384Z * [new branch] gh/ezyang/550/head -> origin/gh/ezyang/550/head 2022-03-17T19:04:58.4328772Z * [new branch] gh/ezyang/550/orig -> origin/gh/ezyang/550/orig 2022-03-17T19:04:58.4330274Z * [new branch] gh/ezyang/551/base -> origin/gh/ezyang/551/base 2022-03-17T19:04:58.4331407Z * [new branch] gh/ezyang/551/head -> origin/gh/ezyang/551/head 2022-03-17T19:04:58.4332868Z * [new branch] gh/ezyang/551/orig -> origin/gh/ezyang/551/orig 2022-03-17T19:04:58.4334332Z * [new branch] gh/ezyang/552/base -> origin/gh/ezyang/552/base 2022-03-17T19:04:58.4335652Z * [new branch] gh/ezyang/552/head -> origin/gh/ezyang/552/head 2022-03-17T19:04:58.4337568Z * [new branch] gh/ezyang/552/orig -> origin/gh/ezyang/552/orig 2022-03-17T19:04:58.4339030Z * [new branch] gh/ezyang/553/base -> origin/gh/ezyang/553/base 2022-03-17T19:04:58.4340125Z * [new branch] gh/ezyang/553/head -> origin/gh/ezyang/553/head 2022-03-17T19:04:58.4341735Z * [new branch] gh/ezyang/553/orig -> origin/gh/ezyang/553/orig 2022-03-17T19:04:58.4343214Z * [new branch] gh/ezyang/554/base -> origin/gh/ezyang/554/base 2022-03-17T19:04:58.4344327Z * [new branch] gh/ezyang/554/head -> origin/gh/ezyang/554/head 2022-03-17T19:04:58.4346034Z * [new branch] gh/ezyang/554/orig -> origin/gh/ezyang/554/orig 2022-03-17T19:04:58.4347567Z * [new branch] gh/ezyang/568/base -> origin/gh/ezyang/568/base 2022-03-17T19:04:58.4348869Z * [new branch] gh/ezyang/568/head -> origin/gh/ezyang/568/head 2022-03-17T19:04:58.4350335Z * [new branch] gh/ezyang/568/orig -> origin/gh/ezyang/568/orig 2022-03-17T19:04:58.4352130Z * [new branch] gh/ezyang/573/base -> origin/gh/ezyang/573/base 2022-03-17T19:04:58.4353257Z * [new branch] gh/ezyang/573/head -> origin/gh/ezyang/573/head 2022-03-17T19:04:58.4354613Z * [new branch] gh/ezyang/573/orig -> origin/gh/ezyang/573/orig 2022-03-17T19:04:58.4356391Z * [new branch] gh/ezyang/574/base -> origin/gh/ezyang/574/base 2022-03-17T19:04:58.4357840Z * [new branch] gh/ezyang/574/head -> origin/gh/ezyang/574/head 2022-03-17T19:04:58.4359135Z * [new branch] gh/ezyang/574/orig -> origin/gh/ezyang/574/orig 2022-03-17T19:04:58.4360616Z * [new branch] gh/ezyang/575/base -> origin/gh/ezyang/575/base 2022-03-17T19:04:58.4362106Z * [new branch] gh/ezyang/575/head -> origin/gh/ezyang/575/head 2022-03-17T19:04:58.4363524Z * [new branch] gh/ezyang/575/orig -> origin/gh/ezyang/575/orig 2022-03-17T19:04:58.4364951Z * [new branch] gh/ezyang/576/base -> origin/gh/ezyang/576/base 2022-03-17T19:04:58.4366330Z * [new branch] gh/ezyang/576/head -> origin/gh/ezyang/576/head 2022-03-17T19:04:58.4367651Z * [new branch] gh/ezyang/576/orig -> origin/gh/ezyang/576/orig 2022-03-17T19:04:58.4369315Z * [new branch] gh/ezyang/577/base -> origin/gh/ezyang/577/base 2022-03-17T19:04:58.4370685Z * [new branch] gh/ezyang/577/head -> origin/gh/ezyang/577/head 2022-03-17T19:04:58.4372058Z * [new branch] gh/ezyang/577/orig -> origin/gh/ezyang/577/orig 2022-03-17T19:04:58.4374442Z * [new branch] gh/ezyang/590/base -> origin/gh/ezyang/590/base 2022-03-17T19:04:58.4375749Z * [new branch] gh/ezyang/590/head -> origin/gh/ezyang/590/head 2022-03-17T19:04:58.4377092Z * [new branch] gh/ezyang/590/orig -> origin/gh/ezyang/590/orig 2022-03-17T19:04:58.4378825Z * [new branch] gh/ezyang/604/base -> origin/gh/ezyang/604/base 2022-03-17T19:04:58.4380181Z * [new branch] gh/ezyang/604/head -> origin/gh/ezyang/604/head 2022-03-17T19:04:58.4381579Z * [new branch] gh/ezyang/604/orig -> origin/gh/ezyang/604/orig 2022-03-17T19:04:58.4383233Z * [new branch] gh/ezyang/608/base -> origin/gh/ezyang/608/base 2022-03-17T19:04:58.4384600Z * [new branch] gh/ezyang/608/head -> origin/gh/ezyang/608/head 2022-03-17T19:04:58.4385923Z * [new branch] gh/ezyang/608/orig -> origin/gh/ezyang/608/orig 2022-03-17T19:04:58.4387564Z * [new branch] gh/ezyang/613/base -> origin/gh/ezyang/613/base 2022-03-17T19:04:58.4388906Z * [new branch] gh/ezyang/613/head -> origin/gh/ezyang/613/head 2022-03-17T19:04:58.4390494Z * [new branch] gh/ezyang/613/orig -> origin/gh/ezyang/613/orig 2022-03-17T19:04:58.4392186Z * [new branch] gh/ezyang/617/base -> origin/gh/ezyang/617/base 2022-03-17T19:04:58.4396331Z * [new branch] gh/ezyang/617/head -> origin/gh/ezyang/617/head 2022-03-17T19:04:58.4396995Z * [new branch] gh/ezyang/617/orig -> origin/gh/ezyang/617/orig 2022-03-17T19:04:58.4397684Z * [new branch] gh/ezyang/620/base -> origin/gh/ezyang/620/base 2022-03-17T19:04:58.4398350Z * [new branch] gh/ezyang/620/head -> origin/gh/ezyang/620/head 2022-03-17T19:04:58.4399258Z * [new branch] gh/ezyang/620/orig -> origin/gh/ezyang/620/orig 2022-03-17T19:04:58.4401514Z * [new branch] gh/ezyang/624/base -> origin/gh/ezyang/624/base 2022-03-17T19:04:58.4403559Z * [new branch] gh/ezyang/624/head -> origin/gh/ezyang/624/head 2022-03-17T19:04:58.4405610Z * [new branch] gh/ezyang/624/orig -> origin/gh/ezyang/624/orig 2022-03-17T19:04:58.4406905Z * [new branch] gh/ezyang/628/base -> origin/gh/ezyang/628/base 2022-03-17T19:04:58.4408362Z * [new branch] gh/ezyang/628/head -> origin/gh/ezyang/628/head 2022-03-17T19:04:58.4409648Z * [new branch] gh/ezyang/628/orig -> origin/gh/ezyang/628/orig 2022-03-17T19:04:58.4411199Z * [new branch] gh/ezyang/630/base -> origin/gh/ezyang/630/base 2022-03-17T19:04:58.4412623Z * [new branch] gh/ezyang/630/head -> origin/gh/ezyang/630/head 2022-03-17T19:04:58.4413959Z * [new branch] gh/ezyang/630/orig -> origin/gh/ezyang/630/orig 2022-03-17T19:04:58.4415479Z * [new branch] gh/ezyang/631/base -> origin/gh/ezyang/631/base 2022-03-17T19:04:58.4417002Z * [new branch] gh/ezyang/631/head -> origin/gh/ezyang/631/head 2022-03-17T19:04:58.4418395Z * [new branch] gh/ezyang/631/orig -> origin/gh/ezyang/631/orig 2022-03-17T19:04:58.4419931Z * [new branch] gh/ezyang/632/base -> origin/gh/ezyang/632/base 2022-03-17T19:04:58.4421176Z * [new branch] gh/ezyang/632/head -> origin/gh/ezyang/632/head 2022-03-17T19:04:58.4422536Z * [new branch] gh/ezyang/632/orig -> origin/gh/ezyang/632/orig 2022-03-17T19:04:58.4424064Z * [new branch] gh/ezyang/633/base -> origin/gh/ezyang/633/base 2022-03-17T19:04:58.4425357Z * [new branch] gh/ezyang/633/head -> origin/gh/ezyang/633/head 2022-03-17T19:04:58.4426738Z * [new branch] gh/ezyang/633/orig -> origin/gh/ezyang/633/orig 2022-03-17T19:04:58.4428577Z * [new branch] gh/ezyang/635/base -> origin/gh/ezyang/635/base 2022-03-17T19:04:58.4429851Z * [new branch] gh/ezyang/635/head -> origin/gh/ezyang/635/head 2022-03-17T19:04:58.4431291Z * [new branch] gh/ezyang/635/orig -> origin/gh/ezyang/635/orig 2022-03-17T19:04:58.4433219Z * [new branch] gh/ezyang/636/base -> origin/gh/ezyang/636/base 2022-03-17T19:04:58.4434445Z * [new branch] gh/ezyang/636/head -> origin/gh/ezyang/636/head 2022-03-17T19:04:58.4435670Z * [new branch] gh/ezyang/636/orig -> origin/gh/ezyang/636/orig 2022-03-17T19:04:58.4437511Z * [new branch] gh/ezyang/637/base -> origin/gh/ezyang/637/base 2022-03-17T19:04:58.4438725Z * [new branch] gh/ezyang/637/head -> origin/gh/ezyang/637/head 2022-03-17T19:04:58.4440093Z * [new branch] gh/ezyang/637/orig -> origin/gh/ezyang/637/orig 2022-03-17T19:04:58.4442059Z * [new branch] gh/ezyang/638/base -> origin/gh/ezyang/638/base 2022-03-17T19:04:58.4443734Z * [new branch] gh/ezyang/638/head -> origin/gh/ezyang/638/head 2022-03-17T19:04:58.4445047Z * [new branch] gh/ezyang/638/orig -> origin/gh/ezyang/638/orig 2022-03-17T19:04:58.4446841Z * [new branch] gh/ezyang/640/base -> origin/gh/ezyang/640/base 2022-03-17T19:04:58.4448130Z * [new branch] gh/ezyang/640/head -> origin/gh/ezyang/640/head 2022-03-17T19:04:58.4449423Z * [new branch] gh/ezyang/640/orig -> origin/gh/ezyang/640/orig 2022-03-17T19:04:58.4451212Z * [new branch] gh/ezyang/641/base -> origin/gh/ezyang/641/base 2022-03-17T19:04:58.4452430Z * [new branch] gh/ezyang/641/head -> origin/gh/ezyang/641/head 2022-03-17T19:04:58.4453835Z * [new branch] gh/ezyang/641/orig -> origin/gh/ezyang/641/orig 2022-03-17T19:04:58.4455897Z * [new branch] gh/ezyang/644/base -> origin/gh/ezyang/644/base 2022-03-17T19:04:58.4457067Z * [new branch] gh/ezyang/644/head -> origin/gh/ezyang/644/head 2022-03-17T19:04:58.4458523Z * [new branch] gh/ezyang/644/orig -> origin/gh/ezyang/644/orig 2022-03-17T19:04:58.4460183Z * [new branch] gh/ezyang/648/base -> origin/gh/ezyang/648/base 2022-03-17T19:04:58.4461159Z * [new branch] gh/ezyang/648/head -> origin/gh/ezyang/648/head 2022-03-17T19:04:58.4462611Z * [new branch] gh/ezyang/648/orig -> origin/gh/ezyang/648/orig 2022-03-17T19:04:58.4464578Z * [new branch] gh/ezyang/650/base -> origin/gh/ezyang/650/base 2022-03-17T19:04:58.4466101Z * [new branch] gh/ezyang/650/head -> origin/gh/ezyang/650/head 2022-03-17T19:04:58.4467658Z * [new branch] gh/ezyang/650/orig -> origin/gh/ezyang/650/orig 2022-03-17T19:04:58.4469118Z * [new branch] gh/ezyang/671/base -> origin/gh/ezyang/671/base 2022-03-17T19:04:58.4470545Z * [new branch] gh/ezyang/671/head -> origin/gh/ezyang/671/head 2022-03-17T19:04:58.4471904Z * [new branch] gh/ezyang/671/orig -> origin/gh/ezyang/671/orig 2022-03-17T19:04:58.4473498Z * [new branch] gh/ezyang/672/base -> origin/gh/ezyang/672/base 2022-03-17T19:04:58.4474994Z * [new branch] gh/ezyang/672/head -> origin/gh/ezyang/672/head 2022-03-17T19:04:58.4476420Z * [new branch] gh/ezyang/672/orig -> origin/gh/ezyang/672/orig 2022-03-17T19:04:58.4478149Z * [new branch] gh/ezyang/754/base -> origin/gh/ezyang/754/base 2022-03-17T19:04:58.4479499Z * [new branch] gh/ezyang/754/head -> origin/gh/ezyang/754/head 2022-03-17T19:04:58.4481223Z * [new branch] gh/ezyang/780/head -> origin/gh/ezyang/780/head 2022-03-17T19:04:58.4483226Z * [new branch] gh/ezyang/811/base -> origin/gh/ezyang/811/base 2022-03-17T19:04:58.4484648Z * [new branch] gh/ezyang/811/orig -> origin/gh/ezyang/811/orig 2022-03-17T19:04:58.4486321Z * [new branch] gh/ezyang/838/base -> origin/gh/ezyang/838/base 2022-03-17T19:04:58.4487722Z * [new branch] gh/ezyang/838/head -> origin/gh/ezyang/838/head 2022-03-17T19:04:58.4488985Z * [new branch] gh/ezyang/838/orig -> origin/gh/ezyang/838/orig 2022-03-17T19:04:58.4490904Z * [new branch] gh/ezyang/870/base -> origin/gh/ezyang/870/base 2022-03-17T19:04:58.4492558Z * [new branch] gh/ezyang/870/head -> origin/gh/ezyang/870/head 2022-03-17T19:04:58.4493770Z * [new branch] gh/ezyang/870/orig -> origin/gh/ezyang/870/orig 2022-03-17T19:04:58.4495539Z * [new branch] gh/ezyang/906/base -> origin/gh/ezyang/906/base 2022-03-17T19:04:58.4496963Z * [new branch] gh/ezyang/906/head -> origin/gh/ezyang/906/head 2022-03-17T19:04:58.4498348Z * [new branch] gh/ezyang/906/orig -> origin/gh/ezyang/906/orig 2022-03-17T19:04:58.4500066Z * [new branch] gh/ezyang/915/base -> origin/gh/ezyang/915/base 2022-03-17T19:04:58.4501642Z * [new branch] gh/ezyang/915/head -> origin/gh/ezyang/915/head 2022-03-17T19:04:58.4502867Z * [new branch] gh/ezyang/915/orig -> origin/gh/ezyang/915/orig 2022-03-17T19:04:58.4504582Z * [new branch] gh/ezyang/932/base -> origin/gh/ezyang/932/base 2022-03-17T19:04:58.4505882Z * [new branch] gh/ezyang/932/head -> origin/gh/ezyang/932/head 2022-03-17T19:04:58.4507283Z * [new branch] gh/ezyang/932/orig -> origin/gh/ezyang/932/orig 2022-03-17T19:04:58.4508939Z * [new branch] gh/ezyang/933/base -> origin/gh/ezyang/933/base 2022-03-17T19:04:58.4510318Z * [new branch] gh/ezyang/933/head -> origin/gh/ezyang/933/head 2022-03-17T19:04:58.4511689Z * [new branch] gh/ezyang/933/orig -> origin/gh/ezyang/933/orig 2022-03-17T19:04:58.4513367Z * [new branch] gh/ezyang/952/base -> origin/gh/ezyang/952/base 2022-03-17T19:04:58.4514784Z * [new branch] gh/ezyang/952/head -> origin/gh/ezyang/952/head 2022-03-17T19:04:58.4516173Z * [new branch] gh/ezyang/952/orig -> origin/gh/ezyang/952/orig 2022-03-17T19:04:58.4518025Z * [new branch] gh/ezyang/993/base -> origin/gh/ezyang/993/base 2022-03-17T19:04:58.4519281Z * [new branch] gh/ezyang/993/head -> origin/gh/ezyang/993/head 2022-03-17T19:04:58.4520660Z * [new branch] gh/ezyang/993/orig -> origin/gh/ezyang/993/orig 2022-03-17T19:04:58.4522591Z * [new branch] gh/ezyang/994/base -> origin/gh/ezyang/994/base 2022-03-17T19:04:58.4524121Z * [new branch] gh/ezyang/994/head -> origin/gh/ezyang/994/head 2022-03-17T19:04:58.4525405Z * [new branch] gh/ezyang/994/orig -> origin/gh/ezyang/994/orig 2022-03-17T19:04:58.4527138Z * [new branch] gh/ezyang/995/base -> origin/gh/ezyang/995/base 2022-03-17T19:04:58.4528521Z * [new branch] gh/ezyang/995/head -> origin/gh/ezyang/995/head 2022-03-17T19:04:58.4529799Z * [new branch] gh/ezyang/995/orig -> origin/gh/ezyang/995/orig 2022-03-17T19:04:58.4531441Z * [new branch] gh/ezyang/997/base -> origin/gh/ezyang/997/base 2022-03-17T19:04:58.4532780Z * [new branch] gh/ezyang/997/head -> origin/gh/ezyang/997/head 2022-03-17T19:04:58.4534146Z * [new branch] gh/ezyang/997/orig -> origin/gh/ezyang/997/orig 2022-03-17T19:04:58.4536132Z * [new branch] gh/fduwjj/23/base -> origin/gh/fduwjj/23/base 2022-03-17T19:04:58.4537445Z * [new branch] gh/fduwjj/23/head -> origin/gh/fduwjj/23/head 2022-03-17T19:04:58.4538815Z * [new branch] gh/fduwjj/23/orig -> origin/gh/fduwjj/23/orig 2022-03-17T19:04:58.4540689Z * [new branch] gh/fegin/4/base -> origin/gh/fegin/4/base 2022-03-17T19:04:58.4542141Z * [new branch] gh/fegin/4/head -> origin/gh/fegin/4/head 2022-03-17T19:04:58.4543477Z * [new branch] gh/fegin/4/orig -> origin/gh/fegin/4/orig 2022-03-17T19:04:58.4545360Z * [new branch] gh/firstprayer/1/base -> origin/gh/firstprayer/1/base 2022-03-17T19:04:58.4546697Z * [new branch] gh/firstprayer/1/head -> origin/gh/firstprayer/1/head 2022-03-17T19:04:58.4548022Z * [new branch] gh/firstprayer/1/orig -> origin/gh/firstprayer/1/orig 2022-03-17T19:04:58.4549929Z * [new branch] gh/gamrix/13/base -> origin/gh/gamrix/13/base 2022-03-17T19:04:58.4551235Z * [new branch] gh/gamrix/13/orig -> origin/gh/gamrix/13/orig 2022-03-17T19:04:58.4552877Z * [new branch] gh/gamrix/14/base -> origin/gh/gamrix/14/base 2022-03-17T19:04:58.4554213Z * [new branch] gh/gamrix/14/orig -> origin/gh/gamrix/14/orig 2022-03-17T19:04:58.4555980Z * [new branch] gh/gamrix/15/base -> origin/gh/gamrix/15/base 2022-03-17T19:04:58.4557217Z * [new branch] gh/gamrix/15/orig -> origin/gh/gamrix/15/orig 2022-03-17T19:04:58.4558868Z * [new branch] gh/gamrix/16/base -> origin/gh/gamrix/16/base 2022-03-17T19:04:58.4560213Z * [new branch] gh/gamrix/16/orig -> origin/gh/gamrix/16/orig 2022-03-17T19:04:58.4561984Z * [new branch] gh/gamrix/17/base -> origin/gh/gamrix/17/base 2022-03-17T19:04:58.4563919Z * [new branch] gh/gamrix/17/head -> origin/gh/gamrix/17/head 2022-03-17T19:04:58.4565146Z * [new branch] gh/gamrix/17/orig -> origin/gh/gamrix/17/orig 2022-03-17T19:04:58.4566951Z * [new branch] gh/gamrix/22/base -> origin/gh/gamrix/22/base 2022-03-17T19:04:58.4568487Z * [new branch] gh/gamrix/22/orig -> origin/gh/gamrix/22/orig 2022-03-17T19:04:58.4570162Z * [new branch] gh/gamrix/23/base -> origin/gh/gamrix/23/base 2022-03-17T19:04:58.4571520Z * [new branch] gh/gamrix/23/orig -> origin/gh/gamrix/23/orig 2022-03-17T19:04:58.4573106Z * [new branch] gh/gamrix/25/base -> origin/gh/gamrix/25/base 2022-03-17T19:04:58.4574513Z * [new branch] gh/gamrix/25/head -> origin/gh/gamrix/25/head 2022-03-17T19:04:58.4575894Z * [new branch] gh/gamrix/25/orig -> origin/gh/gamrix/25/orig 2022-03-17T19:04:58.4577555Z * [new branch] gh/gamrix/26/base -> origin/gh/gamrix/26/base 2022-03-17T19:04:58.4578845Z * [new branch] gh/gamrix/26/head -> origin/gh/gamrix/26/head 2022-03-17T19:04:58.4580205Z * [new branch] gh/gamrix/26/orig -> origin/gh/gamrix/26/orig 2022-03-17T19:04:58.4581846Z * [new branch] gh/gamrix/30/base -> origin/gh/gamrix/30/base 2022-03-17T19:04:58.4583181Z * [new branch] gh/gamrix/30/head -> origin/gh/gamrix/30/head 2022-03-17T19:04:58.4584456Z * [new branch] gh/gamrix/30/orig -> origin/gh/gamrix/30/orig 2022-03-17T19:04:58.4586070Z * [new branch] gh/gamrix/43/base -> origin/gh/gamrix/43/base 2022-03-17T19:04:58.4587436Z * [new branch] gh/gamrix/43/head -> origin/gh/gamrix/43/head 2022-03-17T19:04:58.4588703Z * [new branch] gh/gamrix/43/orig -> origin/gh/gamrix/43/orig 2022-03-17T19:04:58.4590476Z * [new branch] gh/gamrix/44/base -> origin/gh/gamrix/44/base 2022-03-17T19:04:58.4592275Z * [new branch] gh/gamrix/44/head -> origin/gh/gamrix/44/head 2022-03-17T19:04:58.4593384Z * [new branch] gh/gamrix/44/orig -> origin/gh/gamrix/44/orig 2022-03-17T19:04:58.4594899Z * [new branch] gh/gamrix/46/base -> origin/gh/gamrix/46/base 2022-03-17T19:04:58.4596137Z * [new branch] gh/gamrix/46/head -> origin/gh/gamrix/46/head 2022-03-17T19:04:58.4597364Z * [new branch] gh/gamrix/46/orig -> origin/gh/gamrix/46/orig 2022-03-17T19:04:58.4598911Z * [new branch] gh/gamrix/47/base -> origin/gh/gamrix/47/base 2022-03-17T19:04:58.4600193Z * [new branch] gh/gamrix/47/head -> origin/gh/gamrix/47/head 2022-03-17T19:04:58.4601490Z * [new branch] gh/gamrix/47/orig -> origin/gh/gamrix/47/orig 2022-03-17T19:04:58.4603528Z * [new branch] gh/gamrix/49/base -> origin/gh/gamrix/49/base 2022-03-17T19:04:58.4604774Z * [new branch] gh/gamrix/49/head -> origin/gh/gamrix/49/head 2022-03-17T19:04:58.4606112Z * [new branch] gh/gamrix/49/orig -> origin/gh/gamrix/49/orig 2022-03-17T19:04:58.4607891Z * [new branch] gh/gamrix/50/base -> origin/gh/gamrix/50/base 2022-03-17T19:04:58.4609041Z * [new branch] gh/gamrix/50/head -> origin/gh/gamrix/50/head 2022-03-17T19:04:58.4610523Z * [new branch] gh/gamrix/50/orig -> origin/gh/gamrix/50/orig 2022-03-17T19:04:58.4612189Z * [new branch] gh/gamrix/51/base -> origin/gh/gamrix/51/base 2022-03-17T19:04:58.4613512Z * [new branch] gh/gamrix/51/head -> origin/gh/gamrix/51/head 2022-03-17T19:04:58.4614764Z * [new branch] gh/gamrix/51/orig -> origin/gh/gamrix/51/orig 2022-03-17T19:04:58.4616724Z * [new branch] gh/gchanan/110/head -> origin/gh/gchanan/110/head 2022-03-17T19:04:58.4618331Z * [new branch] gh/gchanan/166/base -> origin/gh/gchanan/166/base 2022-03-17T19:04:58.4619657Z * [new branch] gh/gchanan/166/head -> origin/gh/gchanan/166/head 2022-03-17T19:04:58.4621019Z * [new branch] gh/gchanan/166/orig -> origin/gh/gchanan/166/orig 2022-03-17T19:04:58.4622676Z * [new branch] gh/gchanan/178/base -> origin/gh/gchanan/178/base 2022-03-17T19:04:58.4624095Z * [new branch] gh/gchanan/178/head -> origin/gh/gchanan/178/head 2022-03-17T19:04:58.4625381Z * [new branch] gh/gchanan/178/orig -> origin/gh/gchanan/178/orig 2022-03-17T19:04:58.4627228Z * [new branch] gh/gchanan/268/base -> origin/gh/gchanan/268/base 2022-03-17T19:04:58.4628425Z * [new branch] gh/gchanan/268/head -> origin/gh/gchanan/268/head 2022-03-17T19:04:58.4629817Z * [new branch] gh/gchanan/268/orig -> origin/gh/gchanan/268/orig 2022-03-17T19:04:58.4631669Z * [new branch] gh/gchanan/269/base -> origin/gh/gchanan/269/base 2022-03-17T19:04:58.4632898Z * [new branch] gh/gchanan/269/head -> origin/gh/gchanan/269/head 2022-03-17T19:04:58.4634326Z * [new branch] gh/gchanan/269/orig -> origin/gh/gchanan/269/orig 2022-03-17T19:04:58.4636132Z * [new branch] gh/gchanan/302/base -> origin/gh/gchanan/302/base 2022-03-17T19:04:58.4637376Z * [new branch] gh/gchanan/302/head -> origin/gh/gchanan/302/head 2022-03-17T19:04:58.4638771Z * [new branch] gh/gchanan/302/orig -> origin/gh/gchanan/302/orig 2022-03-17T19:04:58.4640406Z * [new branch] gh/gchanan/329/base -> origin/gh/gchanan/329/base 2022-03-17T19:04:58.4642181Z * [new branch] gh/gchanan/329/head -> origin/gh/gchanan/329/head 2022-03-17T19:04:58.4643617Z * [new branch] gh/gchanan/329/orig -> origin/gh/gchanan/329/orig 2022-03-17T19:04:58.4645196Z * [new branch] gh/gchanan/330/base -> origin/gh/gchanan/330/base 2022-03-17T19:04:58.4646531Z * [new branch] gh/gchanan/330/head -> origin/gh/gchanan/330/head 2022-03-17T19:04:58.4647925Z * [new branch] gh/gchanan/330/orig -> origin/gh/gchanan/330/orig 2022-03-17T19:04:58.4650226Z * [new branch] gh/gchanan/336/base -> origin/gh/gchanan/336/base 2022-03-17T19:04:58.4651214Z * [new branch] gh/gchanan/336/head -> origin/gh/gchanan/336/head 2022-03-17T19:04:58.4652447Z * [new branch] gh/gchanan/336/orig -> origin/gh/gchanan/336/orig 2022-03-17T19:04:58.4654274Z * [new branch] gh/gchanan/45/base -> origin/gh/gchanan/45/base 2022-03-17T19:04:58.4655609Z * [new branch] gh/gchanan/45/head -> origin/gh/gchanan/45/head 2022-03-17T19:04:58.4656758Z * [new branch] gh/gchanan/45/orig -> origin/gh/gchanan/45/orig 2022-03-17T19:04:58.4658468Z * [new branch] gh/gchanan/64/base -> origin/gh/gchanan/64/base 2022-03-17T19:04:58.4659826Z * [new branch] gh/gchanan/64/head -> origin/gh/gchanan/64/head 2022-03-17T19:04:58.4661162Z * [new branch] gh/gchanan/64/orig -> origin/gh/gchanan/64/orig 2022-03-17T19:04:58.4662708Z * [new branch] gh/gchanan/65/base -> origin/gh/gchanan/65/base 2022-03-17T19:04:58.4664123Z * [new branch] gh/gchanan/65/head -> origin/gh/gchanan/65/head 2022-03-17T19:04:58.4665418Z * [new branch] gh/gchanan/65/orig -> origin/gh/gchanan/65/orig 2022-03-17T19:04:58.4667113Z * [new branch] gh/gchanan/70/base -> origin/gh/gchanan/70/base 2022-03-17T19:04:58.4668493Z * [new branch] gh/gchanan/70/head -> origin/gh/gchanan/70/head 2022-03-17T19:04:58.4669827Z * [new branch] gh/gchanan/70/orig -> origin/gh/gchanan/70/orig 2022-03-17T19:04:58.4671335Z * [new branch] gh/gchanan/71/base -> origin/gh/gchanan/71/base 2022-03-17T19:04:58.4672746Z * [new branch] gh/gchanan/71/head -> origin/gh/gchanan/71/head 2022-03-17T19:04:58.4674031Z * [new branch] gh/gchanan/71/orig -> origin/gh/gchanan/71/orig 2022-03-17T19:04:58.4675537Z * [new branch] gh/gchanan/72/base -> origin/gh/gchanan/72/base 2022-03-17T19:04:58.4676844Z * [new branch] gh/gchanan/72/head -> origin/gh/gchanan/72/head 2022-03-17T19:04:58.4678208Z * [new branch] gh/gchanan/72/orig -> origin/gh/gchanan/72/orig 2022-03-17T19:04:58.4679959Z * [new branch] gh/gchanan/80/base -> origin/gh/gchanan/80/base 2022-03-17T19:04:58.4681356Z * [new branch] gh/gchanan/80/head -> origin/gh/gchanan/80/head 2022-03-17T19:04:58.4685147Z * [new branch] gh/gchanan/80/orig -> origin/gh/gchanan/80/orig 2022-03-17T19:04:58.4685496Z * [new branch] gh/gcramer23/1/base -> origin/gh/gcramer23/1/base 2022-03-17T19:04:58.4686212Z * [new branch] gh/gcramer23/1/head -> origin/gh/gcramer23/1/head 2022-03-17T19:04:58.4687563Z * [new branch] gh/gcramer23/1/orig -> origin/gh/gcramer23/1/orig 2022-03-17T19:04:58.4689238Z * [new branch] gh/gcramer23/12/base -> origin/gh/gcramer23/12/base 2022-03-17T19:04:58.4690525Z * [new branch] gh/gcramer23/12/orig -> origin/gh/gcramer23/12/orig 2022-03-17T19:04:58.4692210Z * [new branch] gh/gcramer23/13/base -> origin/gh/gcramer23/13/base 2022-03-17T19:04:58.4693539Z * [new branch] gh/gcramer23/13/head -> origin/gh/gcramer23/13/head 2022-03-17T19:04:58.4694874Z * [new branch] gh/gcramer23/13/orig -> origin/gh/gcramer23/13/orig 2022-03-17T19:04:58.4696544Z * [new branch] gh/gcramer23/16/base -> origin/gh/gcramer23/16/base 2022-03-17T19:04:58.4698016Z * [new branch] gh/gcramer23/16/orig -> origin/gh/gcramer23/16/orig 2022-03-17T19:04:58.4699618Z * [new branch] gh/gcramer23/17/base -> origin/gh/gcramer23/17/base 2022-03-17T19:04:58.4700866Z * [new branch] gh/gcramer23/17/orig -> origin/gh/gcramer23/17/orig 2022-03-17T19:04:58.4702670Z * [new branch] gh/gcramer23/19/base -> origin/gh/gcramer23/19/base 2022-03-17T19:04:58.4703989Z * [new branch] gh/gcramer23/19/head -> origin/gh/gcramer23/19/head 2022-03-17T19:04:58.4705733Z * [new branch] gh/gcramer23/2/base -> origin/gh/gcramer23/2/base 2022-03-17T19:04:58.4707148Z * [new branch] gh/gcramer23/2/head -> origin/gh/gcramer23/2/head 2022-03-17T19:04:58.4708484Z * [new branch] gh/gcramer23/2/orig -> origin/gh/gcramer23/2/orig 2022-03-17T19:04:58.4709961Z * [new branch] gh/gcramer23/20/base -> origin/gh/gcramer23/20/base 2022-03-17T19:04:58.4711304Z * [new branch] gh/gcramer23/20/head -> origin/gh/gcramer23/20/head 2022-03-17T19:04:58.4712616Z * [new branch] gh/gcramer23/20/orig -> origin/gh/gcramer23/20/orig 2022-03-17T19:04:58.4714361Z * [new branch] gh/gcramer23/21/base -> origin/gh/gcramer23/21/base 2022-03-17T19:04:58.4715792Z * [new branch] gh/gcramer23/21/orig -> origin/gh/gcramer23/21/orig 2022-03-17T19:04:58.4717651Z * [new branch] gh/gcramer23/23/base -> origin/gh/gcramer23/23/base 2022-03-17T19:04:58.4718857Z * [new branch] gh/gcramer23/23/head -> origin/gh/gcramer23/23/head 2022-03-17T19:04:58.4720113Z * [new branch] gh/gcramer23/23/orig -> origin/gh/gcramer23/23/orig 2022-03-17T19:04:58.4721950Z * [new branch] gh/gcramer23/5/base -> origin/gh/gcramer23/5/base 2022-03-17T19:04:58.4724290Z * [new branch] gh/gcramer23/5/orig -> origin/gh/gcramer23/5/orig 2022-03-17T19:04:58.4725684Z * [new branch] gh/gcramer23/6/base -> origin/gh/gcramer23/6/base 2022-03-17T19:04:58.4727009Z * [new branch] gh/gcramer23/6/head -> origin/gh/gcramer23/6/head 2022-03-17T19:04:58.4728923Z * [new branch] gh/gcramer23/8/base -> origin/gh/gcramer23/8/base 2022-03-17T19:04:58.4730303Z * [new branch] gh/gcramer23/8/head -> origin/gh/gcramer23/8/head 2022-03-17T19:04:58.4731684Z * [new branch] gh/gcramer23/8/orig -> origin/gh/gcramer23/8/orig 2022-03-17T19:04:58.4733406Z * [new branch] gh/george-qi/19/base -> origin/gh/george-qi/19/base 2022-03-17T19:04:58.4734720Z * [new branch] gh/george-qi/19/head -> origin/gh/george-qi/19/head 2022-03-17T19:04:58.4736077Z * [new branch] gh/george-qi/19/orig -> origin/gh/george-qi/19/orig 2022-03-17T19:04:58.4737772Z * [new branch] gh/george-qi/20/base -> origin/gh/george-qi/20/base 2022-03-17T19:04:58.4738997Z * [new branch] gh/george-qi/20/head -> origin/gh/george-qi/20/head 2022-03-17T19:04:58.4740368Z * [new branch] gh/george-qi/20/orig -> origin/gh/george-qi/20/orig 2022-03-17T19:04:58.4742062Z * [new branch] gh/george-qi/21/base -> origin/gh/george-qi/21/base 2022-03-17T19:04:58.4743475Z * [new branch] gh/george-qi/21/head -> origin/gh/george-qi/21/head 2022-03-17T19:04:58.4744759Z * [new branch] gh/george-qi/21/orig -> origin/gh/george-qi/21/orig 2022-03-17T19:04:58.4746223Z * [new branch] gh/george-qi/22/base -> origin/gh/george-qi/22/base 2022-03-17T19:04:58.4747551Z * [new branch] gh/george-qi/22/head -> origin/gh/george-qi/22/head 2022-03-17T19:04:58.4748942Z * [new branch] gh/george-qi/22/orig -> origin/gh/george-qi/22/orig 2022-03-17T19:04:58.4750362Z * [new branch] gh/george-qi/23/base -> origin/gh/george-qi/23/base 2022-03-17T19:04:58.4751685Z * [new branch] gh/george-qi/23/head -> origin/gh/george-qi/23/head 2022-03-17T19:04:58.4753069Z * [new branch] gh/george-qi/23/orig -> origin/gh/george-qi/23/orig 2022-03-17T19:04:58.4754535Z * [new branch] gh/george-qi/24/base -> origin/gh/george-qi/24/base 2022-03-17T19:04:58.4755866Z * [new branch] gh/george-qi/24/head -> origin/gh/george-qi/24/head 2022-03-17T19:04:58.4757119Z * [new branch] gh/george-qi/24/orig -> origin/gh/george-qi/24/orig 2022-03-17T19:04:58.4758621Z * [new branch] gh/george-qi/25/base -> origin/gh/george-qi/25/base 2022-03-17T19:04:58.4759924Z * [new branch] gh/george-qi/25/head -> origin/gh/george-qi/25/head 2022-03-17T19:04:58.4761279Z * [new branch] gh/george-qi/25/orig -> origin/gh/george-qi/25/orig 2022-03-17T19:04:58.4763330Z * [new branch] gh/glaringlee/31/base -> origin/gh/glaringlee/31/base 2022-03-17T19:04:58.4764707Z * [new branch] gh/glaringlee/31/head -> origin/gh/glaringlee/31/head 2022-03-17T19:04:58.4765997Z * [new branch] gh/glaringlee/31/orig -> origin/gh/glaringlee/31/orig 2022-03-17T19:04:58.4767817Z * [new branch] gh/gqchen/1/base -> origin/gh/gqchen/1/base 2022-03-17T19:04:58.4769424Z * [new branch] gh/gqchen/1/head -> origin/gh/gqchen/1/head 2022-03-17T19:04:58.4770782Z * [new branch] gh/gqchen/1/orig -> origin/gh/gqchen/1/orig 2022-03-17T19:04:58.4772656Z * [new branch] gh/haichuan-fb/2/base -> origin/gh/haichuan-fb/2/base 2022-03-17T19:04:58.4773952Z * [new branch] gh/haichuan-fb/2/orig -> origin/gh/haichuan-fb/2/orig 2022-03-17T19:04:58.4775921Z * [new branch] gh/hameerabbasi/5/base -> origin/gh/hameerabbasi/5/base 2022-03-17T19:04:58.4777182Z * [new branch] gh/hameerabbasi/5/orig -> origin/gh/hameerabbasi/5/orig 2022-03-17T19:04:58.4779033Z * [new branch] gh/heitorschueroff/14/base -> origin/gh/heitorschueroff/14/base 2022-03-17T19:04:58.4780414Z * [new branch] gh/heitorschueroff/14/head -> origin/gh/heitorschueroff/14/head 2022-03-17T19:04:58.4781712Z * [new branch] gh/heitorschueroff/14/orig -> origin/gh/heitorschueroff/14/orig 2022-03-17T19:04:58.4783302Z * [new branch] gh/heitorschueroff/17/base -> origin/gh/heitorschueroff/17/base 2022-03-17T19:04:58.4784727Z * [new branch] gh/heitorschueroff/17/head -> origin/gh/heitorschueroff/17/head 2022-03-17T19:04:58.4786667Z * [new branch] gh/heitorschueroff/17/orig -> origin/gh/heitorschueroff/17/orig 2022-03-17T19:04:58.4788199Z * [new branch] gh/heitorschueroff/18/base -> origin/gh/heitorschueroff/18/base 2022-03-17T19:04:58.4789474Z * [new branch] gh/heitorschueroff/18/head -> origin/gh/heitorschueroff/18/head 2022-03-17T19:04:58.4790965Z * [new branch] gh/heitorschueroff/18/orig -> origin/gh/heitorschueroff/18/orig 2022-03-17T19:04:58.4792494Z * [new branch] gh/heitorschueroff/19/base -> origin/gh/heitorschueroff/19/base 2022-03-17T19:04:58.4793727Z * [new branch] gh/heitorschueroff/19/head -> origin/gh/heitorschueroff/19/head 2022-03-17T19:04:58.4795025Z * [new branch] gh/heitorschueroff/19/orig -> origin/gh/heitorschueroff/19/orig 2022-03-17T19:04:58.4797018Z * [new branch] gh/houseroad/9/base -> origin/gh/houseroad/9/base 2022-03-17T19:04:58.4798328Z * [new branch] gh/houseroad/9/orig -> origin/gh/houseroad/9/orig 2022-03-17T19:04:58.4800313Z * [new branch] gh/huiguoo/16/base -> origin/gh/huiguoo/16/base 2022-03-17T19:04:58.4801623Z * [new branch] gh/huiguoo/16/head -> origin/gh/huiguoo/16/head 2022-03-17T19:04:58.4803269Z * [new branch] gh/huiguoo/16/orig -> origin/gh/huiguoo/16/orig 2022-03-17T19:04:58.4804918Z * [new branch] gh/huiguoo/3/base -> origin/gh/huiguoo/3/base 2022-03-17T19:04:58.4806151Z * [new branch] gh/huiguoo/3/orig -> origin/gh/huiguoo/3/orig 2022-03-17T19:04:58.4807828Z * [new branch] gh/huiguoo/39/base -> origin/gh/huiguoo/39/base 2022-03-17T19:04:58.4809258Z * [new branch] gh/huiguoo/39/head -> origin/gh/huiguoo/39/head 2022-03-17T19:04:58.4810543Z * [new branch] gh/huiguoo/39/orig -> origin/gh/huiguoo/39/orig 2022-03-17T19:04:58.4812348Z * [new branch] gh/huiguoo/4/base -> origin/gh/huiguoo/4/base 2022-03-17T19:04:58.4813655Z * [new branch] gh/huiguoo/4/orig -> origin/gh/huiguoo/4/orig 2022-03-17T19:04:58.4815232Z * [new branch] gh/huiguoo/40/base -> origin/gh/huiguoo/40/base 2022-03-17T19:04:58.4816638Z * [new branch] gh/huiguoo/40/head -> origin/gh/huiguoo/40/head 2022-03-17T19:04:58.4817864Z * [new branch] gh/huiguoo/40/orig -> origin/gh/huiguoo/40/orig 2022-03-17T19:04:58.4819622Z * [new branch] gh/huiguoo/41/base -> origin/gh/huiguoo/41/base 2022-03-17T19:04:58.4820798Z * [new branch] gh/huiguoo/41/head -> origin/gh/huiguoo/41/head 2022-03-17T19:04:58.4822188Z * [new branch] gh/huiguoo/41/orig -> origin/gh/huiguoo/41/orig 2022-03-17T19:04:58.4823900Z * [new branch] gh/huiguoo/42/base -> origin/gh/huiguoo/42/base 2022-03-17T19:04:58.4824955Z * [new branch] gh/huiguoo/42/head -> origin/gh/huiguoo/42/head 2022-03-17T19:04:58.4826224Z * [new branch] gh/huiguoo/42/orig -> origin/gh/huiguoo/42/orig 2022-03-17T19:04:58.4828035Z * [new branch] gh/huiguoo/50/base -> origin/gh/huiguoo/50/base 2022-03-17T19:04:58.4829555Z * [new branch] gh/huiguoo/50/head -> origin/gh/huiguoo/50/head 2022-03-17T19:04:58.4830769Z * [new branch] gh/huiguoo/50/orig -> origin/gh/huiguoo/50/orig 2022-03-17T19:04:58.4832643Z * [new branch] gh/huiguoo/51/base -> origin/gh/huiguoo/51/base 2022-03-17T19:04:58.4833976Z * [new branch] gh/huiguoo/51/head -> origin/gh/huiguoo/51/head 2022-03-17T19:04:58.4835355Z * [new branch] gh/huiguoo/51/orig -> origin/gh/huiguoo/51/orig 2022-03-17T19:04:58.4836975Z * [new branch] gh/huiguoo/54/base -> origin/gh/huiguoo/54/base 2022-03-17T19:04:58.4838318Z * [new branch] gh/huiguoo/54/head -> origin/gh/huiguoo/54/head 2022-03-17T19:04:58.4839596Z * [new branch] gh/huiguoo/54/orig -> origin/gh/huiguoo/54/orig 2022-03-17T19:04:58.4841363Z * [new branch] gh/huiguoo/57/base -> origin/gh/huiguoo/57/base 2022-03-17T19:04:58.4842820Z * [new branch] gh/huiguoo/57/head -> origin/gh/huiguoo/57/head 2022-03-17T19:04:58.4845301Z * [new branch] gh/huiguoo/57/orig -> origin/gh/huiguoo/57/orig 2022-03-17T19:04:58.4845884Z * [new branch] gh/huiguoo/58/base -> origin/gh/huiguoo/58/base 2022-03-17T19:04:58.4847151Z * [new branch] gh/huiguoo/58/head -> origin/gh/huiguoo/58/head 2022-03-17T19:04:58.4849149Z * [new branch] gh/huiguoo/58/orig -> origin/gh/huiguoo/58/orig 2022-03-17T19:04:58.4850349Z * [new branch] gh/husthyc/10/base -> origin/gh/husthyc/10/base 2022-03-17T19:04:58.4851859Z * [new branch] gh/husthyc/10/head -> origin/gh/husthyc/10/head 2022-03-17T19:04:58.4853096Z * [new branch] gh/husthyc/10/orig -> origin/gh/husthyc/10/orig 2022-03-17T19:04:58.4854980Z * [new branch] gh/husthyc/2/base -> origin/gh/husthyc/2/base 2022-03-17T19:04:58.4856216Z * [new branch] gh/husthyc/2/orig -> origin/gh/husthyc/2/orig 2022-03-17T19:04:58.4857919Z * [new branch] gh/husthyc/22/base -> origin/gh/husthyc/22/base 2022-03-17T19:04:58.4859210Z * [new branch] gh/husthyc/22/head -> origin/gh/husthyc/22/head 2022-03-17T19:04:58.4860929Z * [new branch] gh/husthyc/25/base -> origin/gh/husthyc/25/base 2022-03-17T19:04:58.4862340Z * [new branch] gh/husthyc/25/orig -> origin/gh/husthyc/25/orig 2022-03-17T19:04:58.4864068Z * [new branch] gh/husthyc/26/base -> origin/gh/husthyc/26/base 2022-03-17T19:04:58.4865317Z * [new branch] gh/husthyc/26/orig -> origin/gh/husthyc/26/orig 2022-03-17T19:04:58.4867176Z * [new branch] gh/husthyc/27/base -> origin/gh/husthyc/27/base 2022-03-17T19:04:58.4868383Z * [new branch] gh/husthyc/27/orig -> origin/gh/husthyc/27/orig 2022-03-17T19:04:58.4870160Z * [new branch] gh/husthyc/3/orig -> origin/gh/husthyc/3/orig 2022-03-17T19:04:58.4871799Z * [new branch] gh/husthyc/32/base -> origin/gh/husthyc/32/base 2022-03-17T19:04:58.4873182Z * [new branch] gh/husthyc/32/head -> origin/gh/husthyc/32/head 2022-03-17T19:04:58.4874580Z * [new branch] gh/husthyc/32/orig -> origin/gh/husthyc/32/orig 2022-03-17T19:04:58.4876113Z * [new branch] gh/husthyc/33/base -> origin/gh/husthyc/33/base 2022-03-17T19:04:58.4877471Z * [new branch] gh/husthyc/33/head -> origin/gh/husthyc/33/head 2022-03-17T19:04:58.4878648Z * [new branch] gh/husthyc/33/orig -> origin/gh/husthyc/33/orig 2022-03-17T19:04:58.4880166Z * [new branch] gh/husthyc/34/base -> origin/gh/husthyc/34/base 2022-03-17T19:04:58.4881365Z * [new branch] gh/husthyc/34/head -> origin/gh/husthyc/34/head 2022-03-17T19:04:58.4883871Z * [new branch] gh/husthyc/34/orig -> origin/gh/husthyc/34/orig 2022-03-17T19:04:58.4885551Z * [new branch] gh/husthyc/4/base -> origin/gh/husthyc/4/base 2022-03-17T19:04:58.4887255Z * [new branch] gh/husthyc/5/base -> origin/gh/husthyc/5/base 2022-03-17T19:04:58.4888666Z * [new branch] gh/husthyc/6/base -> origin/gh/husthyc/6/base 2022-03-17T19:04:58.4890196Z * [new branch] gh/husthyc/7/base -> origin/gh/husthyc/7/base 2022-03-17T19:04:58.4891630Z * [new branch] gh/husthyc/8/base -> origin/gh/husthyc/8/base 2022-03-17T19:04:58.4893439Z * [new branch] gh/husthyc/9/base -> origin/gh/husthyc/9/base 2022-03-17T19:04:58.4895192Z * [new branch] gh/hx89/10/base -> origin/gh/hx89/10/base 2022-03-17T19:04:58.4896594Z * [new branch] gh/hx89/10/head -> origin/gh/hx89/10/head 2022-03-17T19:04:58.4897985Z * [new branch] gh/hx89/10/orig -> origin/gh/hx89/10/orig 2022-03-17T19:04:58.4899660Z * [new branch] gh/hx89/11/base -> origin/gh/hx89/11/base 2022-03-17T19:04:58.4900990Z * [new branch] gh/hx89/11/head -> origin/gh/hx89/11/head 2022-03-17T19:04:58.4902214Z * [new branch] gh/hx89/11/orig -> origin/gh/hx89/11/orig 2022-03-17T19:04:58.4904063Z * [new branch] gh/hx89/13/base -> origin/gh/hx89/13/base 2022-03-17T19:04:58.4905384Z * [new branch] gh/hx89/13/head -> origin/gh/hx89/13/head 2022-03-17T19:04:58.4906516Z * [new branch] gh/hx89/13/orig -> origin/gh/hx89/13/orig 2022-03-17T19:04:58.4908292Z * [new branch] gh/hx89/8/base -> origin/gh/hx89/8/base 2022-03-17T19:04:58.4909658Z * [new branch] gh/hx89/8/head -> origin/gh/hx89/8/head 2022-03-17T19:04:58.4911057Z * [new branch] gh/hx89/8/orig -> origin/gh/hx89/8/orig 2022-03-17T19:04:58.4912843Z * [new branch] gh/ifedan/114/base -> origin/gh/ifedan/114/base 2022-03-17T19:04:58.4914125Z * [new branch] gh/ifedan/114/head -> origin/gh/ifedan/114/head 2022-03-17T19:04:58.4915873Z * [new branch] gh/ifedan/115/base -> origin/gh/ifedan/115/base 2022-03-17T19:04:58.4917296Z * [new branch] gh/ifedan/115/head -> origin/gh/ifedan/115/head 2022-03-17T19:04:58.4918791Z * [new branch] gh/ifedan/116/base -> origin/gh/ifedan/116/base 2022-03-17T19:04:58.4919994Z * [new branch] gh/ifedan/116/head -> origin/gh/ifedan/116/head 2022-03-17T19:04:58.4921685Z * [new branch] gh/ifedan/117/base -> origin/gh/ifedan/117/base 2022-03-17T19:04:58.4923204Z * [new branch] gh/ifedan/117/head -> origin/gh/ifedan/117/head 2022-03-17T19:04:58.4926187Z * [new branch] gh/ifedan/68/base -> origin/gh/ifedan/68/base 2022-03-17T19:04:58.4927013Z * [new branch] gh/ifedan/68/head -> origin/gh/ifedan/68/head 2022-03-17T19:04:58.4927744Z * [new branch] gh/ifedan/68/orig -> origin/gh/ifedan/68/orig 2022-03-17T19:04:58.4929045Z * [new branch] gh/ifedan/69/base -> origin/gh/ifedan/69/base 2022-03-17T19:04:58.4930079Z * [new branch] gh/ifedan/69/head -> origin/gh/ifedan/69/head 2022-03-17T19:04:58.4931314Z * [new branch] gh/ifedan/69/orig -> origin/gh/ifedan/69/orig 2022-03-17T19:04:58.4933791Z * [new branch] gh/ifedan/70/base -> origin/gh/ifedan/70/base 2022-03-17T19:04:58.4934278Z * [new branch] gh/ifedan/70/head -> origin/gh/ifedan/70/head 2022-03-17T19:04:58.4935513Z * [new branch] gh/ifedan/70/orig -> origin/gh/ifedan/70/orig 2022-03-17T19:04:58.4937661Z * [new branch] gh/ifedan/71/base -> origin/gh/ifedan/71/base 2022-03-17T19:04:58.4938693Z * [new branch] gh/ifedan/71/head -> origin/gh/ifedan/71/head 2022-03-17T19:04:58.4940399Z * [new branch] gh/ifedan/71/orig -> origin/gh/ifedan/71/orig 2022-03-17T19:04:58.4941629Z * [new branch] gh/ifedan/72/base -> origin/gh/ifedan/72/base 2022-03-17T19:04:58.4943255Z * [new branch] gh/ifedan/72/head -> origin/gh/ifedan/72/head 2022-03-17T19:04:58.4944379Z * [new branch] gh/ifedan/72/orig -> origin/gh/ifedan/72/orig 2022-03-17T19:04:58.4946152Z * [new branch] gh/ifedan/73/base -> origin/gh/ifedan/73/base 2022-03-17T19:04:58.4947243Z * [new branch] gh/ifedan/73/head -> origin/gh/ifedan/73/head 2022-03-17T19:04:58.4948544Z * [new branch] gh/ifedan/73/orig -> origin/gh/ifedan/73/orig 2022-03-17T19:04:58.4950304Z * [new branch] gh/ifedan/74/base -> origin/gh/ifedan/74/base 2022-03-17T19:04:58.4951466Z * [new branch] gh/ifedan/74/head -> origin/gh/ifedan/74/head 2022-03-17T19:04:58.4952732Z * [new branch] gh/ifedan/74/orig -> origin/gh/ifedan/74/orig 2022-03-17T19:04:58.4954982Z * [new branch] gh/ilia-cher/39/base -> origin/gh/ilia-cher/39/base 2022-03-17T19:04:58.4956175Z * [new branch] gh/ilia-cher/39/orig -> origin/gh/ilia-cher/39/orig 2022-03-17T19:04:58.4958067Z * [new branch] gh/ilia-cher/50/base -> origin/gh/ilia-cher/50/base 2022-03-17T19:04:58.4959220Z * [new branch] gh/ilia-cher/50/head -> origin/gh/ilia-cher/50/head 2022-03-17T19:04:58.4960872Z * [new branch] gh/ilia-cher/51/base -> origin/gh/ilia-cher/51/base 2022-03-17T19:04:58.4962132Z * [new branch] gh/ilia-cher/51/head -> origin/gh/ilia-cher/51/head 2022-03-17T19:04:58.4964321Z * [new branch] gh/ilia-cher/53/base -> origin/gh/ilia-cher/53/base 2022-03-17T19:04:58.4965457Z * [new branch] gh/ilia-cher/53/head -> origin/gh/ilia-cher/53/head 2022-03-17T19:04:58.4966716Z * [new branch] gh/ilia-cher/53/orig -> origin/gh/ilia-cher/53/orig 2022-03-17T19:04:58.4968615Z * [new branch] gh/ilia-cher/83/base -> origin/gh/ilia-cher/83/base 2022-03-17T19:04:58.4969821Z * [new branch] gh/ilia-cher/83/head -> origin/gh/ilia-cher/83/head 2022-03-17T19:04:58.4971679Z * [new branch] gh/ilia-cher/84/base -> origin/gh/ilia-cher/84/base 2022-03-17T19:04:58.4972907Z * [new branch] gh/ilia-cher/84/head -> origin/gh/ilia-cher/84/head 2022-03-17T19:04:58.4974167Z * [new branch] gh/ilia-cher/84/orig -> origin/gh/ilia-cher/84/orig 2022-03-17T19:04:58.4976228Z * [new branch] gh/ilia-cher/86/base -> origin/gh/ilia-cher/86/base 2022-03-17T19:04:58.4977827Z * [new branch] gh/ilia-cher/86/head -> origin/gh/ilia-cher/86/head 2022-03-17T19:04:58.4979132Z * [new branch] gh/ilia-cher/86/orig -> origin/gh/ilia-cher/86/orig 2022-03-17T19:04:58.4981104Z * [new branch] gh/iramazanli/10/base -> origin/gh/iramazanli/10/base 2022-03-17T19:04:58.4982367Z * [new branch] gh/iramazanli/10/head -> origin/gh/iramazanli/10/head 2022-03-17T19:04:58.4983731Z * [new branch] gh/iramazanli/10/orig -> origin/gh/iramazanli/10/orig 2022-03-17T19:04:58.4985566Z * [new branch] gh/iramazanli/11/base -> origin/gh/iramazanli/11/base 2022-03-17T19:04:58.4986983Z * [new branch] gh/iramazanli/11/head -> origin/gh/iramazanli/11/head 2022-03-17T19:04:58.4988169Z * [new branch] gh/iramazanli/11/orig -> origin/gh/iramazanli/11/orig 2022-03-17T19:04:58.4990060Z * [new branch] gh/iramazanli/12/base -> origin/gh/iramazanli/12/base 2022-03-17T19:04:58.4991331Z * [new branch] gh/iramazanli/12/head -> origin/gh/iramazanli/12/head 2022-03-17T19:04:58.4992561Z * [new branch] gh/iramazanli/12/orig -> origin/gh/iramazanli/12/orig 2022-03-17T19:04:58.4994436Z * [new branch] gh/iramazanli/8/base -> origin/gh/iramazanli/8/base 2022-03-17T19:04:58.4995627Z * [new branch] gh/iramazanli/8/head -> origin/gh/iramazanli/8/head 2022-03-17T19:04:58.4997267Z * [new branch] gh/iramazanli/8/orig -> origin/gh/iramazanli/8/orig 2022-03-17T19:04:58.4998587Z * [new branch] gh/iramazanli/9/base -> origin/gh/iramazanli/9/base 2022-03-17T19:04:58.4999977Z * [new branch] gh/iramazanli/9/head -> origin/gh/iramazanli/9/head 2022-03-17T19:04:58.5001244Z * [new branch] gh/iramazanli/9/orig -> origin/gh/iramazanli/9/orig 2022-03-17T19:04:58.5003546Z * [new branch] gh/iseeyuan/1/base -> origin/gh/iseeyuan/1/base 2022-03-17T19:04:58.5005398Z * [new branch] gh/iseeyuan/10/base -> origin/gh/iseeyuan/10/base 2022-03-17T19:04:58.5006556Z * [new branch] gh/iseeyuan/10/head -> origin/gh/iseeyuan/10/head 2022-03-17T19:04:58.5007873Z * [new branch] gh/iseeyuan/10/orig -> origin/gh/iseeyuan/10/orig 2022-03-17T19:04:58.5009907Z * [new branch] gh/iseeyuan/101/base -> origin/gh/iseeyuan/101/base 2022-03-17T19:04:58.5011231Z * [new branch] gh/iseeyuan/101/head -> origin/gh/iseeyuan/101/head 2022-03-17T19:04:58.5012391Z * [new branch] gh/iseeyuan/101/orig -> origin/gh/iseeyuan/101/orig 2022-03-17T19:04:58.5014279Z * [new branch] gh/iseeyuan/110/base -> origin/gh/iseeyuan/110/base 2022-03-17T19:04:58.5015579Z * [new branch] gh/iseeyuan/110/head -> origin/gh/iseeyuan/110/head 2022-03-17T19:04:58.5016859Z * [new branch] gh/iseeyuan/110/orig -> origin/gh/iseeyuan/110/orig 2022-03-17T19:04:58.5018739Z * [new branch] gh/iseeyuan/112/base -> origin/gh/iseeyuan/112/base 2022-03-17T19:04:58.5020289Z * [new branch] gh/iseeyuan/112/head -> origin/gh/iseeyuan/112/head 2022-03-17T19:04:58.5021505Z * [new branch] gh/iseeyuan/112/orig -> origin/gh/iseeyuan/112/orig 2022-03-17T19:04:58.5023580Z * [new branch] gh/iseeyuan/114/base -> origin/gh/iseeyuan/114/base 2022-03-17T19:04:58.5024754Z * [new branch] gh/iseeyuan/114/head -> origin/gh/iseeyuan/114/head 2022-03-17T19:04:58.5025883Z * [new branch] gh/iseeyuan/114/orig -> origin/gh/iseeyuan/114/orig 2022-03-17T19:04:58.5027586Z * [new branch] gh/iseeyuan/116/base -> origin/gh/iseeyuan/116/base 2022-03-17T19:04:58.5029546Z * [new branch] gh/iseeyuan/116/head -> origin/gh/iseeyuan/116/head 2022-03-17T19:04:58.5030597Z * [new branch] gh/iseeyuan/116/orig -> origin/gh/iseeyuan/116/orig 2022-03-17T19:04:58.5032530Z * [new branch] gh/iseeyuan/118/base -> origin/gh/iseeyuan/118/base 2022-03-17T19:04:58.5033680Z * [new branch] gh/iseeyuan/118/head -> origin/gh/iseeyuan/118/head 2022-03-17T19:04:58.5035051Z * [new branch] gh/iseeyuan/118/orig -> origin/gh/iseeyuan/118/orig 2022-03-17T19:04:58.5036700Z * [new branch] gh/iseeyuan/119/base -> origin/gh/iseeyuan/119/base 2022-03-17T19:04:58.5037832Z * [new branch] gh/iseeyuan/119/head -> origin/gh/iseeyuan/119/head 2022-03-17T19:04:58.5039421Z * [new branch] gh/iseeyuan/119/orig -> origin/gh/iseeyuan/119/orig 2022-03-17T19:04:58.5040970Z * [new branch] gh/iseeyuan/120/base -> origin/gh/iseeyuan/120/base 2022-03-17T19:04:58.5042439Z * [new branch] gh/iseeyuan/120/head -> origin/gh/iseeyuan/120/head 2022-03-17T19:04:58.5044882Z * [new branch] gh/iseeyuan/120/orig -> origin/gh/iseeyuan/120/orig 2022-03-17T19:04:58.5046557Z * [new branch] gh/iseeyuan/122/base -> origin/gh/iseeyuan/122/base 2022-03-17T19:04:58.5047945Z * [new branch] gh/iseeyuan/122/head -> origin/gh/iseeyuan/122/head 2022-03-17T19:04:58.5049120Z * [new branch] gh/iseeyuan/122/orig -> origin/gh/iseeyuan/122/orig 2022-03-17T19:04:58.5050726Z * [new branch] gh/iseeyuan/125/base -> origin/gh/iseeyuan/125/base 2022-03-17T19:04:58.5052409Z * [new branch] gh/iseeyuan/125/head -> origin/gh/iseeyuan/125/head 2022-03-17T19:04:58.5053574Z * [new branch] gh/iseeyuan/125/orig -> origin/gh/iseeyuan/125/orig 2022-03-17T19:04:58.5055342Z * [new branch] gh/iseeyuan/128/base -> origin/gh/iseeyuan/128/base 2022-03-17T19:04:58.5056633Z * [new branch] gh/iseeyuan/128/head -> origin/gh/iseeyuan/128/head 2022-03-17T19:04:58.5057960Z * [new branch] gh/iseeyuan/128/orig -> origin/gh/iseeyuan/128/orig 2022-03-17T19:04:58.5059680Z * [new branch] gh/iseeyuan/130/base -> origin/gh/iseeyuan/130/base 2022-03-17T19:04:58.5061075Z * [new branch] gh/iseeyuan/130/head -> origin/gh/iseeyuan/130/head 2022-03-17T19:04:58.5062384Z * [new branch] gh/iseeyuan/130/orig -> origin/gh/iseeyuan/130/orig 2022-03-17T19:04:58.5064135Z * [new branch] gh/iseeyuan/133/base -> origin/gh/iseeyuan/133/base 2022-03-17T19:04:58.5065438Z * [new branch] gh/iseeyuan/133/head -> origin/gh/iseeyuan/133/head 2022-03-17T19:04:58.5066775Z * [new branch] gh/iseeyuan/133/orig -> origin/gh/iseeyuan/133/orig 2022-03-17T19:04:58.5068426Z * [new branch] gh/iseeyuan/134/base -> origin/gh/iseeyuan/134/base 2022-03-17T19:04:58.5069801Z * [new branch] gh/iseeyuan/134/head -> origin/gh/iseeyuan/134/head 2022-03-17T19:04:58.5071165Z * [new branch] gh/iseeyuan/134/orig -> origin/gh/iseeyuan/134/orig 2022-03-17T19:04:58.5072606Z * [new branch] gh/iseeyuan/18/base -> origin/gh/iseeyuan/18/base 2022-03-17T19:04:58.5074096Z * [new branch] gh/iseeyuan/18/head -> origin/gh/iseeyuan/18/head 2022-03-17T19:04:58.5075399Z * [new branch] gh/iseeyuan/18/orig -> origin/gh/iseeyuan/18/orig 2022-03-17T19:04:58.5077130Z * [new branch] gh/iseeyuan/2/base -> origin/gh/iseeyuan/2/base 2022-03-17T19:04:58.5078651Z * [new branch] gh/iseeyuan/2/head -> origin/gh/iseeyuan/2/head 2022-03-17T19:04:58.5080018Z * [new branch] gh/iseeyuan/2/orig -> origin/gh/iseeyuan/2/orig 2022-03-17T19:04:58.5081662Z * [new branch] gh/iseeyuan/20/base -> origin/gh/iseeyuan/20/base 2022-03-17T19:04:58.5083312Z * [new branch] gh/iseeyuan/20/head -> origin/gh/iseeyuan/20/head 2022-03-17T19:04:58.5084664Z * [new branch] gh/iseeyuan/20/orig -> origin/gh/iseeyuan/20/orig 2022-03-17T19:04:58.5086311Z * [new branch] gh/iseeyuan/22/base -> origin/gh/iseeyuan/22/base 2022-03-17T19:04:58.5087568Z * [new branch] gh/iseeyuan/22/head -> origin/gh/iseeyuan/22/head 2022-03-17T19:04:58.5088874Z * [new branch] gh/iseeyuan/22/orig -> origin/gh/iseeyuan/22/orig 2022-03-17T19:04:58.5090741Z * [new branch] gh/iseeyuan/25/base -> origin/gh/iseeyuan/25/base 2022-03-17T19:04:58.5092016Z * [new branch] gh/iseeyuan/25/head -> origin/gh/iseeyuan/25/head 2022-03-17T19:04:58.5093546Z * [new branch] gh/iseeyuan/25/orig -> origin/gh/iseeyuan/25/orig 2022-03-17T19:04:58.5095295Z * [new branch] gh/iseeyuan/29/base -> origin/gh/iseeyuan/29/base 2022-03-17T19:04:58.5096392Z * [new branch] gh/iseeyuan/29/head -> origin/gh/iseeyuan/29/head 2022-03-17T19:04:58.5097848Z * [new branch] gh/iseeyuan/29/orig -> origin/gh/iseeyuan/29/orig 2022-03-17T19:04:58.5099413Z * [new branch] gh/iseeyuan/3/base -> origin/gh/iseeyuan/3/base 2022-03-17T19:04:58.5100723Z * [new branch] gh/iseeyuan/3/head -> origin/gh/iseeyuan/3/head 2022-03-17T19:04:58.5102031Z * [new branch] gh/iseeyuan/3/orig -> origin/gh/iseeyuan/3/orig 2022-03-17T19:04:58.5103789Z * [new branch] gh/iseeyuan/30/base -> origin/gh/iseeyuan/30/base 2022-03-17T19:04:58.5105163Z * [new branch] gh/iseeyuan/30/head -> origin/gh/iseeyuan/30/head 2022-03-17T19:04:58.5106432Z * [new branch] gh/iseeyuan/30/orig -> origin/gh/iseeyuan/30/orig 2022-03-17T19:04:58.5108168Z * [new branch] gh/iseeyuan/38/base -> origin/gh/iseeyuan/38/base 2022-03-17T19:04:58.5109442Z * [new branch] gh/iseeyuan/38/head -> origin/gh/iseeyuan/38/head 2022-03-17T19:04:58.5110823Z * [new branch] gh/iseeyuan/38/orig -> origin/gh/iseeyuan/38/orig 2022-03-17T19:04:58.5112193Z * [new branch] gh/iseeyuan/4/base -> origin/gh/iseeyuan/4/base 2022-03-17T19:04:58.5113191Z * [new branch] gh/iseeyuan/4/head -> origin/gh/iseeyuan/4/head 2022-03-17T19:04:58.5115035Z * [new branch] gh/iseeyuan/4/orig -> origin/gh/iseeyuan/4/orig 2022-03-17T19:04:58.5116628Z * [new branch] gh/iseeyuan/48/base -> origin/gh/iseeyuan/48/base 2022-03-17T19:04:58.5117813Z * [new branch] gh/iseeyuan/48/head -> origin/gh/iseeyuan/48/head 2022-03-17T19:04:58.5119257Z * [new branch] gh/iseeyuan/48/orig -> origin/gh/iseeyuan/48/orig 2022-03-17T19:04:58.5120735Z * [new branch] gh/iseeyuan/5/base -> origin/gh/iseeyuan/5/base 2022-03-17T19:04:58.5122006Z * [new branch] gh/iseeyuan/5/head -> origin/gh/iseeyuan/5/head 2022-03-17T19:04:58.5124483Z * [new branch] gh/iseeyuan/5/orig -> origin/gh/iseeyuan/5/orig 2022-03-17T19:04:58.5126039Z * [new branch] gh/iseeyuan/52/base -> origin/gh/iseeyuan/52/base 2022-03-17T19:04:58.5127437Z * [new branch] gh/iseeyuan/52/head -> origin/gh/iseeyuan/52/head 2022-03-17T19:04:58.5128649Z * [new branch] gh/iseeyuan/52/orig -> origin/gh/iseeyuan/52/orig 2022-03-17T19:04:58.5130804Z * [new branch] gh/iseeyuan/56/base -> origin/gh/iseeyuan/56/base 2022-03-17T19:04:58.5131995Z * [new branch] gh/iseeyuan/56/head -> origin/gh/iseeyuan/56/head 2022-03-17T19:04:58.5133444Z * [new branch] gh/iseeyuan/56/orig -> origin/gh/iseeyuan/56/orig 2022-03-17T19:04:58.5134938Z * [new branch] gh/iseeyuan/6/base -> origin/gh/iseeyuan/6/base 2022-03-17T19:04:58.5136555Z * [new branch] gh/iseeyuan/6/head -> origin/gh/iseeyuan/6/head 2022-03-17T19:04:58.5137780Z * [new branch] gh/iseeyuan/6/orig -> origin/gh/iseeyuan/6/orig 2022-03-17T19:04:58.5139590Z * [new branch] gh/iseeyuan/62/base -> origin/gh/iseeyuan/62/base 2022-03-17T19:04:58.5140962Z * [new branch] gh/iseeyuan/62/head -> origin/gh/iseeyuan/62/head 2022-03-17T19:04:58.5142406Z * [new branch] gh/iseeyuan/62/orig -> origin/gh/iseeyuan/62/orig 2022-03-17T19:04:58.5144218Z * [new branch] gh/iseeyuan/65/base -> origin/gh/iseeyuan/65/base 2022-03-17T19:04:58.5145682Z * [new branch] gh/iseeyuan/65/head -> origin/gh/iseeyuan/65/head 2022-03-17T19:04:58.5147302Z * [new branch] gh/iseeyuan/65/orig -> origin/gh/iseeyuan/65/orig 2022-03-17T19:04:58.5148515Z * [new branch] gh/iseeyuan/66/base -> origin/gh/iseeyuan/66/base 2022-03-17T19:04:58.5150007Z * [new branch] gh/iseeyuan/66/head -> origin/gh/iseeyuan/66/head 2022-03-17T19:04:58.5151466Z * [new branch] gh/iseeyuan/66/orig -> origin/gh/iseeyuan/66/orig 2022-03-17T19:04:58.5153128Z * [new branch] gh/iseeyuan/73/base -> origin/gh/iseeyuan/73/base 2022-03-17T19:04:58.5154350Z * [new branch] gh/iseeyuan/73/head -> origin/gh/iseeyuan/73/head 2022-03-17T19:04:58.5155764Z * [new branch] gh/iseeyuan/73/orig -> origin/gh/iseeyuan/73/orig 2022-03-17T19:04:58.5157565Z * [new branch] gh/iseeyuan/77/base -> origin/gh/iseeyuan/77/base 2022-03-17T19:04:58.5158726Z * [new branch] gh/iseeyuan/77/head -> origin/gh/iseeyuan/77/head 2022-03-17T19:04:58.5160188Z * [new branch] gh/iseeyuan/77/orig -> origin/gh/iseeyuan/77/orig 2022-03-17T19:04:58.5161982Z * [new branch] gh/iseeyuan/81/base -> origin/gh/iseeyuan/81/base 2022-03-17T19:04:58.5163438Z * [new branch] gh/iseeyuan/81/head -> origin/gh/iseeyuan/81/head 2022-03-17T19:04:58.5165006Z * [new branch] gh/iseeyuan/81/orig -> origin/gh/iseeyuan/81/orig 2022-03-17T19:04:58.5166408Z * [new branch] gh/iseeyuan/82/base -> origin/gh/iseeyuan/82/base 2022-03-17T19:04:58.5167851Z * [new branch] gh/iseeyuan/82/head -> origin/gh/iseeyuan/82/head 2022-03-17T19:04:58.5169074Z * [new branch] gh/iseeyuan/82/orig -> origin/gh/iseeyuan/82/orig 2022-03-17T19:04:58.5170824Z * [new branch] gh/iseeyuan/83/base -> origin/gh/iseeyuan/83/base 2022-03-17T19:04:58.5172201Z * [new branch] gh/iseeyuan/83/head -> origin/gh/iseeyuan/83/head 2022-03-17T19:04:58.5173385Z * [new branch] gh/iseeyuan/83/orig -> origin/gh/iseeyuan/83/orig 2022-03-17T19:04:58.5175373Z * [new branch] gh/iseeyuan/90/base -> origin/gh/iseeyuan/90/base 2022-03-17T19:04:58.5176584Z * [new branch] gh/iseeyuan/90/head -> origin/gh/iseeyuan/90/head 2022-03-17T19:04:58.5178188Z * [new branch] gh/iseeyuan/90/orig -> origin/gh/iseeyuan/90/orig 2022-03-17T19:04:58.5179879Z * [new branch] gh/iseeyuan/93/base -> origin/gh/iseeyuan/93/base 2022-03-17T19:04:58.5181048Z * [new branch] gh/iseeyuan/93/head -> origin/gh/iseeyuan/93/head 2022-03-17T19:04:58.5182512Z * [new branch] gh/iseeyuan/93/orig -> origin/gh/iseeyuan/93/orig 2022-03-17T19:04:58.5184219Z * [new branch] gh/iseeyuan/95/base -> origin/gh/iseeyuan/95/base 2022-03-17T19:04:58.5185396Z * [new branch] gh/iseeyuan/95/head -> origin/gh/iseeyuan/95/head 2022-03-17T19:04:58.5186592Z * [new branch] gh/iseeyuan/95/orig -> origin/gh/iseeyuan/95/orig 2022-03-17T19:04:58.5188066Z * [new branch] gh/iseeyuan/97/base -> origin/gh/iseeyuan/97/base 2022-03-17T19:04:58.5189435Z * [new branch] gh/iseeyuan/97/head -> origin/gh/iseeyuan/97/head 2022-03-17T19:04:58.5190878Z * [new branch] gh/iseeyuan/97/orig -> origin/gh/iseeyuan/97/orig 2022-03-17T19:04:58.5192478Z * [new branch] gh/iseeyuan/98/base -> origin/gh/iseeyuan/98/base 2022-03-17T19:04:58.5193782Z * [new branch] gh/iseeyuan/98/head -> origin/gh/iseeyuan/98/head 2022-03-17T19:04:58.5195089Z * [new branch] gh/iseeyuan/98/orig -> origin/gh/iseeyuan/98/orig 2022-03-17T19:04:58.5197152Z * [new branch] gh/ivankobzarev/1/base -> origin/gh/ivankobzarev/1/base 2022-03-17T19:04:58.5198558Z * [new branch] gh/ivankobzarev/1/head -> origin/gh/ivankobzarev/1/head 2022-03-17T19:04:58.5200347Z * [new branch] gh/ivankobzarev/102/base -> origin/gh/ivankobzarev/102/base 2022-03-17T19:04:58.5201671Z * [new branch] gh/ivankobzarev/102/head -> origin/gh/ivankobzarev/102/head 2022-03-17T19:04:58.5203960Z * [new branch] gh/ivankobzarev/102/orig -> origin/gh/ivankobzarev/102/orig 2022-03-17T19:04:58.5206118Z * [new branch] gh/ivankobzarev/103/base -> origin/gh/ivankobzarev/103/base 2022-03-17T19:04:58.5207108Z * [new branch] gh/ivankobzarev/103/head -> origin/gh/ivankobzarev/103/head 2022-03-17T19:04:58.5208501Z * [new branch] gh/ivankobzarev/103/orig -> origin/gh/ivankobzarev/103/orig 2022-03-17T19:04:58.5210105Z * [new branch] gh/ivankobzarev/104/base -> origin/gh/ivankobzarev/104/base 2022-03-17T19:04:58.5211429Z * [new branch] gh/ivankobzarev/104/head -> origin/gh/ivankobzarev/104/head 2022-03-17T19:04:58.5212588Z * [new branch] gh/ivankobzarev/104/orig -> origin/gh/ivankobzarev/104/orig 2022-03-17T19:04:58.5214493Z * [new branch] gh/ivankobzarev/106/base -> origin/gh/ivankobzarev/106/base 2022-03-17T19:04:58.5215675Z * [new branch] gh/ivankobzarev/106/head -> origin/gh/ivankobzarev/106/head 2022-03-17T19:04:58.5217143Z * [new branch] gh/ivankobzarev/106/orig -> origin/gh/ivankobzarev/106/orig 2022-03-17T19:04:58.5218914Z * [new branch] gh/ivankobzarev/107/base -> origin/gh/ivankobzarev/107/base 2022-03-17T19:04:58.5220229Z * [new branch] gh/ivankobzarev/107/head -> origin/gh/ivankobzarev/107/head 2022-03-17T19:04:58.5221564Z * [new branch] gh/ivankobzarev/107/orig -> origin/gh/ivankobzarev/107/orig 2022-03-17T19:04:58.5223295Z * [new branch] gh/ivankobzarev/114/base -> origin/gh/ivankobzarev/114/base 2022-03-17T19:04:58.5224634Z * [new branch] gh/ivankobzarev/114/head -> origin/gh/ivankobzarev/114/head 2022-03-17T19:04:58.5225927Z * [new branch] gh/ivankobzarev/114/orig -> origin/gh/ivankobzarev/114/orig 2022-03-17T19:04:58.5227594Z * [new branch] gh/ivankobzarev/115/base -> origin/gh/ivankobzarev/115/base 2022-03-17T19:04:58.5228934Z * [new branch] gh/ivankobzarev/115/head -> origin/gh/ivankobzarev/115/head 2022-03-17T19:04:58.5230276Z * [new branch] gh/ivankobzarev/115/orig -> origin/gh/ivankobzarev/115/orig 2022-03-17T19:04:58.5232033Z * [new branch] gh/ivankobzarev/12/base -> origin/gh/ivankobzarev/12/base 2022-03-17T19:04:58.5233211Z * [new branch] gh/ivankobzarev/12/head -> origin/gh/ivankobzarev/12/head 2022-03-17T19:04:58.5234603Z * [new branch] gh/ivankobzarev/12/orig -> origin/gh/ivankobzarev/12/orig 2022-03-17T19:04:58.5236095Z * [new branch] gh/ivankobzarev/2/base -> origin/gh/ivankobzarev/2/base 2022-03-17T19:04:58.5237450Z * [new branch] gh/ivankobzarev/2/head -> origin/gh/ivankobzarev/2/head 2022-03-17T19:04:58.5239112Z * [new branch] gh/ivankobzarev/3/base -> origin/gh/ivankobzarev/3/base 2022-03-17T19:04:58.5240441Z * [new branch] gh/ivankobzarev/3/head -> origin/gh/ivankobzarev/3/head 2022-03-17T19:04:58.5242177Z * [new branch] gh/ivankobzarev/39/base -> origin/gh/ivankobzarev/39/base 2022-03-17T19:04:58.5243802Z * [new branch] gh/ivankobzarev/39/head -> origin/gh/ivankobzarev/39/head 2022-03-17T19:04:58.5245095Z * [new branch] gh/ivankobzarev/39/orig -> origin/gh/ivankobzarev/39/orig 2022-03-17T19:04:58.5246678Z * [new branch] gh/ivankobzarev/4/base -> origin/gh/ivankobzarev/4/base 2022-03-17T19:04:58.5247966Z * [new branch] gh/ivankobzarev/4/head -> origin/gh/ivankobzarev/4/head 2022-03-17T19:04:58.5249711Z * [new branch] gh/ivankobzarev/41/base -> origin/gh/ivankobzarev/41/base 2022-03-17T19:04:58.5251057Z * [new branch] gh/ivankobzarev/41/head -> origin/gh/ivankobzarev/41/head 2022-03-17T19:04:58.5252583Z * [new branch] gh/ivankobzarev/41/orig -> origin/gh/ivankobzarev/41/orig 2022-03-17T19:04:58.5254132Z * [new branch] gh/ivankobzarev/44/base -> origin/gh/ivankobzarev/44/base 2022-03-17T19:04:58.5255429Z * [new branch] gh/ivankobzarev/44/head -> origin/gh/ivankobzarev/44/head 2022-03-17T19:04:58.5256751Z * [new branch] gh/ivankobzarev/44/orig -> origin/gh/ivankobzarev/44/orig 2022-03-17T19:04:58.5258253Z * [new branch] gh/ivankobzarev/45/base -> origin/gh/ivankobzarev/45/base 2022-03-17T19:04:58.5259565Z * [new branch] gh/ivankobzarev/45/head -> origin/gh/ivankobzarev/45/head 2022-03-17T19:04:58.5260734Z * [new branch] gh/ivankobzarev/45/orig -> origin/gh/ivankobzarev/45/orig 2022-03-17T19:04:58.5262599Z * [new branch] gh/ivankobzarev/49/base -> origin/gh/ivankobzarev/49/base 2022-03-17T19:04:58.5263753Z * [new branch] gh/ivankobzarev/49/head -> origin/gh/ivankobzarev/49/head 2022-03-17T19:04:58.5265090Z * [new branch] gh/ivankobzarev/49/orig -> origin/gh/ivankobzarev/49/orig 2022-03-17T19:04:58.5266660Z * [new branch] gh/ivankobzarev/5/base -> origin/gh/ivankobzarev/5/base 2022-03-17T19:04:58.5267942Z * [new branch] gh/ivankobzarev/5/head -> origin/gh/ivankobzarev/5/head 2022-03-17T19:04:58.5269584Z * [new branch] gh/ivankobzarev/50/base -> origin/gh/ivankobzarev/50/base 2022-03-17T19:04:58.5270959Z * [new branch] gh/ivankobzarev/50/head -> origin/gh/ivankobzarev/50/head 2022-03-17T19:04:58.5272282Z * [new branch] gh/ivankobzarev/50/orig -> origin/gh/ivankobzarev/50/orig 2022-03-17T19:04:58.5274021Z * [new branch] gh/ivankobzarev/51/base -> origin/gh/ivankobzarev/51/base 2022-03-17T19:04:58.5275321Z * [new branch] gh/ivankobzarev/51/head -> origin/gh/ivankobzarev/51/head 2022-03-17T19:04:58.5276627Z * [new branch] gh/ivankobzarev/51/orig -> origin/gh/ivankobzarev/51/orig 2022-03-17T19:04:58.5278439Z * [new branch] gh/ivankobzarev/52/base -> origin/gh/ivankobzarev/52/base 2022-03-17T19:04:58.5279775Z * [new branch] gh/ivankobzarev/52/head -> origin/gh/ivankobzarev/52/head 2022-03-17T19:04:58.5281118Z * [new branch] gh/ivankobzarev/52/orig -> origin/gh/ivankobzarev/52/orig 2022-03-17T19:04:58.5282832Z * [new branch] gh/ivankobzarev/53/base -> origin/gh/ivankobzarev/53/base 2022-03-17T19:04:58.5284271Z * [new branch] gh/ivankobzarev/53/head -> origin/gh/ivankobzarev/53/head 2022-03-17T19:04:58.5286076Z * [new branch] gh/ivankobzarev/53/orig -> origin/gh/ivankobzarev/53/orig 2022-03-17T19:04:58.5287336Z * [new branch] gh/ivankobzarev/54/base -> origin/gh/ivankobzarev/54/base 2022-03-17T19:04:58.5288787Z * [new branch] gh/ivankobzarev/54/head -> origin/gh/ivankobzarev/54/head 2022-03-17T19:04:58.5289984Z * [new branch] gh/ivankobzarev/54/orig -> origin/gh/ivankobzarev/54/orig 2022-03-17T19:04:58.5292157Z * [new branch] gh/ivankobzarev/55/base -> origin/gh/ivankobzarev/55/base 2022-03-17T19:04:58.5293320Z * [new branch] gh/ivankobzarev/55/head -> origin/gh/ivankobzarev/55/head 2022-03-17T19:04:58.5294835Z * [new branch] gh/ivankobzarev/55/orig -> origin/gh/ivankobzarev/55/orig 2022-03-17T19:04:58.5296374Z * [new branch] gh/ivankobzarev/56/base -> origin/gh/ivankobzarev/56/base 2022-03-17T19:04:58.5297546Z * [new branch] gh/ivankobzarev/56/head -> origin/gh/ivankobzarev/56/head 2022-03-17T19:04:58.5299424Z * [new branch] gh/ivankobzarev/57/base -> origin/gh/ivankobzarev/57/base 2022-03-17T19:04:58.5300613Z * [new branch] gh/ivankobzarev/57/head -> origin/gh/ivankobzarev/57/head 2022-03-17T19:04:58.5302248Z * [new branch] gh/ivankobzarev/57/orig -> origin/gh/ivankobzarev/57/orig 2022-03-17T19:04:58.5303831Z * [new branch] gh/ivankobzarev/58/base -> origin/gh/ivankobzarev/58/base 2022-03-17T19:04:58.5304973Z * [new branch] gh/ivankobzarev/58/head -> origin/gh/ivankobzarev/58/head 2022-03-17T19:04:58.5306416Z * [new branch] gh/ivankobzarev/58/orig -> origin/gh/ivankobzarev/58/orig 2022-03-17T19:04:58.5308089Z * [new branch] gh/ivankobzarev/59/base -> origin/gh/ivankobzarev/59/base 2022-03-17T19:04:58.5309506Z * [new branch] gh/ivankobzarev/59/head -> origin/gh/ivankobzarev/59/head 2022-03-17T19:04:58.5310682Z * [new branch] gh/ivankobzarev/59/orig -> origin/gh/ivankobzarev/59/orig 2022-03-17T19:04:58.5312378Z * [new branch] gh/ivankobzarev/6/base -> origin/gh/ivankobzarev/6/base 2022-03-17T19:04:58.5313575Z * [new branch] gh/ivankobzarev/6/head -> origin/gh/ivankobzarev/6/head 2022-03-17T19:04:58.5315422Z * [new branch] gh/ivankobzarev/62/base -> origin/gh/ivankobzarev/62/base 2022-03-17T19:04:58.5316604Z * [new branch] gh/ivankobzarev/62/head -> origin/gh/ivankobzarev/62/head 2022-03-17T19:04:58.5318031Z * [new branch] gh/ivankobzarev/62/orig -> origin/gh/ivankobzarev/62/orig 2022-03-17T19:04:58.5319697Z * [new branch] gh/ivankobzarev/63/base -> origin/gh/ivankobzarev/63/base 2022-03-17T19:04:58.5320910Z * [new branch] gh/ivankobzarev/63/head -> origin/gh/ivankobzarev/63/head 2022-03-17T19:04:58.5322474Z * [new branch] gh/ivankobzarev/63/orig -> origin/gh/ivankobzarev/63/orig 2022-03-17T19:04:58.5324230Z * [new branch] gh/ivankobzarev/64/base -> origin/gh/ivankobzarev/64/base 2022-03-17T19:04:58.5325343Z * [new branch] gh/ivankobzarev/64/head -> origin/gh/ivankobzarev/64/head 2022-03-17T19:04:58.5326766Z * [new branch] gh/ivankobzarev/64/orig -> origin/gh/ivankobzarev/64/orig 2022-03-17T19:04:58.5328418Z * [new branch] gh/ivankobzarev/66/base -> origin/gh/ivankobzarev/66/base 2022-03-17T19:04:58.5329653Z * [new branch] gh/ivankobzarev/66/head -> origin/gh/ivankobzarev/66/head 2022-03-17T19:04:58.5331055Z * [new branch] gh/ivankobzarev/66/orig -> origin/gh/ivankobzarev/66/orig 2022-03-17T19:04:58.5332818Z * [new branch] gh/ivankobzarev/67/base -> origin/gh/ivankobzarev/67/base 2022-03-17T19:04:58.5333942Z * [new branch] gh/ivankobzarev/67/head -> origin/gh/ivankobzarev/67/head 2022-03-17T19:04:58.5335387Z * [new branch] gh/ivankobzarev/67/orig -> origin/gh/ivankobzarev/67/orig 2022-03-17T19:04:58.5337091Z * [new branch] gh/ivankobzarev/68/base -> origin/gh/ivankobzarev/68/base 2022-03-17T19:04:58.5338306Z * [new branch] gh/ivankobzarev/68/head -> origin/gh/ivankobzarev/68/head 2022-03-17T19:04:58.5339849Z * [new branch] gh/ivankobzarev/68/orig -> origin/gh/ivankobzarev/68/orig 2022-03-17T19:04:58.5341341Z * [new branch] gh/ivankobzarev/69/base -> origin/gh/ivankobzarev/69/base 2022-03-17T19:04:58.5342579Z * [new branch] gh/ivankobzarev/69/head -> origin/gh/ivankobzarev/69/head 2022-03-17T19:04:58.5343981Z * [new branch] gh/ivankobzarev/69/orig -> origin/gh/ivankobzarev/69/orig 2022-03-17T19:04:58.5345477Z * [new branch] gh/ivankobzarev/7/base -> origin/gh/ivankobzarev/7/base 2022-03-17T19:04:58.5346723Z * [new branch] gh/ivankobzarev/7/head -> origin/gh/ivankobzarev/7/head 2022-03-17T19:04:58.5348498Z * [new branch] gh/ivankobzarev/70/base -> origin/gh/ivankobzarev/70/base 2022-03-17T19:04:58.5349660Z * [new branch] gh/ivankobzarev/70/head -> origin/gh/ivankobzarev/70/head 2022-03-17T19:04:58.5351134Z * [new branch] gh/ivankobzarev/70/orig -> origin/gh/ivankobzarev/70/orig 2022-03-17T19:04:58.5352878Z * [new branch] gh/ivankobzarev/71/base -> origin/gh/ivankobzarev/71/base 2022-03-17T19:04:58.5354029Z * [new branch] gh/ivankobzarev/71/head -> origin/gh/ivankobzarev/71/head 2022-03-17T19:04:58.5355269Z * [new branch] gh/ivankobzarev/71/orig -> origin/gh/ivankobzarev/71/orig 2022-03-17T19:04:58.5356600Z * [new branch] gh/ivankobzarev/72/base -> origin/gh/ivankobzarev/72/base 2022-03-17T19:04:58.5357865Z * [new branch] gh/ivankobzarev/72/head -> origin/gh/ivankobzarev/72/head 2022-03-17T19:04:58.5359199Z * [new branch] gh/ivankobzarev/72/orig -> origin/gh/ivankobzarev/72/orig 2022-03-17T19:04:58.5365089Z * [new branch] gh/ivankobzarev/73/base -> origin/gh/ivankobzarev/73/base 2022-03-17T19:04:58.5366817Z * [new branch] gh/ivankobzarev/73/head -> origin/gh/ivankobzarev/73/head 2022-03-17T19:04:58.5367889Z * [new branch] gh/ivankobzarev/73/orig -> origin/gh/ivankobzarev/73/orig 2022-03-17T19:04:58.5369748Z * [new branch] gh/ivankobzarev/74/base -> origin/gh/ivankobzarev/74/base 2022-03-17T19:04:58.5371083Z * [new branch] gh/ivankobzarev/74/head -> origin/gh/ivankobzarev/74/head 2022-03-17T19:04:58.5372206Z * [new branch] gh/ivankobzarev/74/orig -> origin/gh/ivankobzarev/74/orig 2022-03-17T19:04:58.5373788Z * [new branch] gh/ivankobzarev/8/base -> origin/gh/ivankobzarev/8/base 2022-03-17T19:04:58.5374937Z * [new branch] gh/ivankobzarev/8/head -> origin/gh/ivankobzarev/8/head 2022-03-17T19:04:58.5376769Z * [new branch] gh/ivankobzarev/83/base -> origin/gh/ivankobzarev/83/base 2022-03-17T19:04:58.5378078Z * [new branch] gh/ivankobzarev/83/head -> origin/gh/ivankobzarev/83/head 2022-03-17T19:04:58.5379411Z * [new branch] gh/ivankobzarev/83/orig -> origin/gh/ivankobzarev/83/orig 2022-03-17T19:04:58.5381278Z * [new branch] gh/ivankobzarev/84/base -> origin/gh/ivankobzarev/84/base 2022-03-17T19:04:58.5382596Z * [new branch] gh/ivankobzarev/84/head -> origin/gh/ivankobzarev/84/head 2022-03-17T19:04:58.5383960Z * [new branch] gh/ivankobzarev/84/orig -> origin/gh/ivankobzarev/84/orig 2022-03-17T19:04:58.5385600Z * [new branch] gh/ivankobzarev/85/base -> origin/gh/ivankobzarev/85/base 2022-03-17T19:04:58.5386958Z * [new branch] gh/ivankobzarev/85/head -> origin/gh/ivankobzarev/85/head 2022-03-17T19:04:58.5388455Z * [new branch] gh/ivankobzarev/85/orig -> origin/gh/ivankobzarev/85/orig 2022-03-17T19:04:58.5390161Z * [new branch] gh/ivankobzarev/86/base -> origin/gh/ivankobzarev/86/base 2022-03-17T19:04:58.5391537Z * [new branch] gh/ivankobzarev/86/head -> origin/gh/ivankobzarev/86/head 2022-03-17T19:04:58.5392846Z * [new branch] gh/ivankobzarev/86/orig -> origin/gh/ivankobzarev/86/orig 2022-03-17T19:04:58.5394610Z * [new branch] gh/ivankobzarev/87/base -> origin/gh/ivankobzarev/87/base 2022-03-17T19:04:58.5395946Z * [new branch] gh/ivankobzarev/87/head -> origin/gh/ivankobzarev/87/head 2022-03-17T19:04:58.5397331Z * [new branch] gh/ivankobzarev/87/orig -> origin/gh/ivankobzarev/87/orig 2022-03-17T19:04:58.5398909Z * [new branch] gh/ivankobzarev/90/base -> origin/gh/ivankobzarev/90/base 2022-03-17T19:04:58.5400248Z * [new branch] gh/ivankobzarev/90/head -> origin/gh/ivankobzarev/90/head 2022-03-17T19:04:58.5401630Z * [new branch] gh/ivankobzarev/90/orig -> origin/gh/ivankobzarev/90/orig 2022-03-17T19:04:58.5403524Z * [new branch] gh/ivankobzarev/94/base -> origin/gh/ivankobzarev/94/base 2022-03-17T19:04:58.5404690Z * [new branch] gh/ivankobzarev/94/head -> origin/gh/ivankobzarev/94/head 2022-03-17T19:04:58.5406043Z * [new branch] gh/ivankobzarev/94/orig -> origin/gh/ivankobzarev/94/orig 2022-03-17T19:04:58.5408015Z * [new branch] gh/ivankobzarev/96/base -> origin/gh/ivankobzarev/96/base 2022-03-17T19:04:58.5409121Z * [new branch] gh/ivankobzarev/96/head -> origin/gh/ivankobzarev/96/head 2022-03-17T19:04:58.5410492Z * [new branch] gh/ivankobzarev/96/orig -> origin/gh/ivankobzarev/96/orig 2022-03-17T19:04:58.5412167Z * [new branch] gh/ivankobzarev/97/base -> origin/gh/ivankobzarev/97/base 2022-03-17T19:04:58.5413600Z * [new branch] gh/ivankobzarev/97/head -> origin/gh/ivankobzarev/97/head 2022-03-17T19:04:58.5414915Z * [new branch] gh/ivankobzarev/97/orig -> origin/gh/ivankobzarev/97/orig 2022-03-17T19:04:58.5416566Z * [new branch] gh/ivankobzarev/98/base -> origin/gh/ivankobzarev/98/base 2022-03-17T19:04:58.5417923Z * [new branch] gh/ivankobzarev/98/head -> origin/gh/ivankobzarev/98/head 2022-03-17T19:04:58.5419273Z * [new branch] gh/ivankobzarev/98/orig -> origin/gh/ivankobzarev/98/orig 2022-03-17T19:04:58.5421262Z * [new branch] gh/ivanyashchuk/46/base -> origin/gh/ivanyashchuk/46/base 2022-03-17T19:04:58.5422446Z * [new branch] gh/ivanyashchuk/46/head -> origin/gh/ivanyashchuk/46/head 2022-03-17T19:04:58.5423583Z * [new branch] gh/ivanyashchuk/46/orig -> origin/gh/ivanyashchuk/46/orig 2022-03-17T19:04:58.5425616Z * [new branch] gh/izdeby/31/base -> origin/gh/izdeby/31/base 2022-03-17T19:04:58.5427410Z * [new branch] gh/izdeby/31/head -> origin/gh/izdeby/31/head 2022-03-17T19:04:58.5428511Z * [new branch] gh/izdeby/31/orig -> origin/gh/izdeby/31/orig 2022-03-17T19:04:58.5430298Z * [new branch] gh/izdeby/32/base -> origin/gh/izdeby/32/base 2022-03-17T19:04:58.5431387Z * [new branch] gh/izdeby/32/head -> origin/gh/izdeby/32/head 2022-03-17T19:04:58.5432787Z * [new branch] gh/izdeby/32/orig -> origin/gh/izdeby/32/orig 2022-03-17T19:04:58.5434902Z * [new branch] gh/izdeby/36/base -> origin/gh/izdeby/36/base 2022-03-17T19:04:58.5436305Z * [new branch] gh/izdeby/36/head -> origin/gh/izdeby/36/head 2022-03-17T19:04:58.5437645Z * [new branch] gh/izdeby/36/orig -> origin/gh/izdeby/36/orig 2022-03-17T19:04:58.5439662Z * [new branch] gh/izdeby/39/base -> origin/gh/izdeby/39/base 2022-03-17T19:04:58.5441134Z * [new branch] gh/izdeby/39/head -> origin/gh/izdeby/39/head 2022-03-17T19:04:58.5443600Z * [new branch] gh/izdeby/39/orig -> origin/gh/izdeby/39/orig 2022-03-17T19:04:58.5445370Z * [new branch] gh/izdeby/64/base -> origin/gh/izdeby/64/base 2022-03-17T19:04:58.5446722Z * [new branch] gh/izdeby/64/head -> origin/gh/izdeby/64/head 2022-03-17T19:04:58.5447985Z * [new branch] gh/izdeby/64/orig -> origin/gh/izdeby/64/orig 2022-03-17T19:04:58.5449881Z * [new branch] gh/izdeby/68/base -> origin/gh/izdeby/68/base 2022-03-17T19:04:58.5451383Z * [new branch] gh/izdeby/68/head -> origin/gh/izdeby/68/head 2022-03-17T19:04:58.5452802Z * [new branch] gh/izdeby/68/orig -> origin/gh/izdeby/68/orig 2022-03-17T19:04:58.5456082Z * [new branch] gh/izdeby/69/base -> origin/gh/izdeby/69/base 2022-03-17T19:04:58.5456675Z * [new branch] gh/izdeby/69/head -> origin/gh/izdeby/69/head 2022-03-17T19:04:58.5457222Z * [new branch] gh/izdeby/69/orig -> origin/gh/izdeby/69/orig 2022-03-17T19:04:58.5459024Z * [new branch] gh/izdeby/74/base -> origin/gh/izdeby/74/base 2022-03-17T19:04:58.5460270Z * [new branch] gh/izdeby/74/head -> origin/gh/izdeby/74/head 2022-03-17T19:04:58.5461784Z * [new branch] gh/izdeby/74/orig -> origin/gh/izdeby/74/orig 2022-03-17T19:04:58.5463748Z * [new branch] gh/izdeby/77/base -> origin/gh/izdeby/77/base 2022-03-17T19:04:58.5464952Z * [new branch] gh/izdeby/77/head -> origin/gh/izdeby/77/head 2022-03-17T19:04:58.5466425Z * [new branch] gh/izdeby/77/orig -> origin/gh/izdeby/77/orig 2022-03-17T19:04:58.5467812Z * [new branch] gh/izdeby/79/base -> origin/gh/izdeby/79/base 2022-03-17T19:04:58.5468901Z * [new branch] gh/izdeby/79/head -> origin/gh/izdeby/79/head 2022-03-17T19:04:58.5470335Z * [new branch] gh/izdeby/79/orig -> origin/gh/izdeby/79/orig 2022-03-17T19:04:58.5472069Z * [new branch] gh/izdeby/82/base -> origin/gh/izdeby/82/base 2022-03-17T19:04:58.5473166Z * [new branch] gh/izdeby/82/orig -> origin/gh/izdeby/82/orig 2022-03-17T19:04:58.5474668Z * [new branch] gh/izdeby/87/base -> origin/gh/izdeby/87/base 2022-03-17T19:04:58.5476017Z * [new branch] gh/izdeby/87/head -> origin/gh/izdeby/87/head 2022-03-17T19:04:58.5477251Z * [new branch] gh/izdeby/87/orig -> origin/gh/izdeby/87/orig 2022-03-17T19:04:58.5478813Z * [new branch] gh/izdeby/88/base -> origin/gh/izdeby/88/base 2022-03-17T19:04:58.5479861Z * [new branch] gh/izdeby/88/head -> origin/gh/izdeby/88/head 2022-03-17T19:04:58.5481241Z * [new branch] gh/izdeby/88/orig -> origin/gh/izdeby/88/orig 2022-03-17T19:04:58.5482894Z * [new branch] gh/izdeby/89/base -> origin/gh/izdeby/89/base 2022-03-17T19:04:58.5484308Z * [new branch] gh/izdeby/89/head -> origin/gh/izdeby/89/head 2022-03-17T19:04:58.5485509Z * [new branch] gh/izdeby/89/orig -> origin/gh/izdeby/89/orig 2022-03-17T19:04:58.5487438Z * [new branch] gh/jaceyca/1/base -> origin/gh/jaceyca/1/base 2022-03-17T19:04:58.5489192Z * [new branch] gh/jaceyca/2/base -> origin/gh/jaceyca/2/base 2022-03-17T19:04:58.5490553Z * [new branch] gh/jaceyca/2/head -> origin/gh/jaceyca/2/head 2022-03-17T19:04:58.5491875Z * [new branch] gh/jaceyca/2/orig -> origin/gh/jaceyca/2/orig 2022-03-17T19:04:58.5493689Z * [new branch] gh/jaceyca/3/base -> origin/gh/jaceyca/3/base 2022-03-17T19:04:58.5495107Z * [new branch] gh/jaceyca/3/head -> origin/gh/jaceyca/3/head 2022-03-17T19:04:58.5496472Z * [new branch] gh/jaceyca/3/orig -> origin/gh/jaceyca/3/orig 2022-03-17T19:04:58.5498183Z * [new branch] gh/jaceyca/4/base -> origin/gh/jaceyca/4/base 2022-03-17T19:04:58.5499474Z * [new branch] gh/jaceyca/4/orig -> origin/gh/jaceyca/4/orig 2022-03-17T19:04:58.5501234Z * [new branch] gh/jaceyca/5/base -> origin/gh/jaceyca/5/base 2022-03-17T19:04:58.5503185Z * [new branch] gh/jacobszwejbka/10/base -> origin/gh/jacobszwejbka/10/base 2022-03-17T19:04:58.5504587Z * [new branch] gh/jacobszwejbka/10/head -> origin/gh/jacobszwejbka/10/head 2022-03-17T19:04:58.5505970Z * [new branch] gh/jacobszwejbka/10/orig -> origin/gh/jacobszwejbka/10/orig 2022-03-17T19:04:58.5507666Z * [new branch] gh/jacobszwejbka/19/base -> origin/gh/jacobszwejbka/19/base 2022-03-17T19:04:58.5509045Z * [new branch] gh/jacobszwejbka/19/head -> origin/gh/jacobszwejbka/19/head 2022-03-17T19:04:58.5510453Z * [new branch] gh/jacobszwejbka/19/orig -> origin/gh/jacobszwejbka/19/orig 2022-03-17T19:04:58.5512267Z * [new branch] gh/jacobszwejbka/2/head -> origin/gh/jacobszwejbka/2/head 2022-03-17T19:04:58.5513434Z * [new branch] gh/jacobszwejbka/2/orig -> origin/gh/jacobszwejbka/2/orig 2022-03-17T19:04:58.5515355Z * [new branch] gh/jacobszwejbka/20/base -> origin/gh/jacobszwejbka/20/base 2022-03-17T19:04:58.5516684Z * [new branch] gh/jacobszwejbka/20/head -> origin/gh/jacobszwejbka/20/head 2022-03-17T19:04:58.5517907Z * [new branch] gh/jacobszwejbka/20/orig -> origin/gh/jacobszwejbka/20/orig 2022-03-17T19:04:58.5519713Z * [new branch] gh/jacobszwejbka/24/base -> origin/gh/jacobszwejbka/24/base 2022-03-17T19:04:58.5521093Z * [new branch] gh/jacobszwejbka/24/head -> origin/gh/jacobszwejbka/24/head 2022-03-17T19:04:58.5522430Z * [new branch] gh/jacobszwejbka/24/orig -> origin/gh/jacobszwejbka/24/orig 2022-03-17T19:04:58.5524338Z * [new branch] gh/jacobszwejbka/25/base -> origin/gh/jacobszwejbka/25/base 2022-03-17T19:04:58.5525811Z * [new branch] gh/jacobszwejbka/25/head -> origin/gh/jacobszwejbka/25/head 2022-03-17T19:04:58.5526892Z * [new branch] gh/jacobszwejbka/25/orig -> origin/gh/jacobszwejbka/25/orig 2022-03-17T19:04:58.5528662Z * [new branch] gh/jacobszwejbka/29/base -> origin/gh/jacobszwejbka/29/base 2022-03-17T19:04:58.5530045Z * [new branch] gh/jacobszwejbka/29/head -> origin/gh/jacobszwejbka/29/head 2022-03-17T19:04:58.5531430Z * [new branch] gh/jacobszwejbka/29/orig -> origin/gh/jacobszwejbka/29/orig 2022-03-17T19:04:58.5533114Z * [new branch] gh/jacobszwejbka/31/base -> origin/gh/jacobszwejbka/31/base 2022-03-17T19:04:58.5534461Z * [new branch] gh/jacobszwejbka/31/head -> origin/gh/jacobszwejbka/31/head 2022-03-17T19:04:58.5535969Z * [new branch] gh/jacobszwejbka/32/base -> origin/gh/jacobszwejbka/32/base 2022-03-17T19:04:58.5537357Z * [new branch] gh/jacobszwejbka/32/head -> origin/gh/jacobszwejbka/32/head 2022-03-17T19:04:58.5538943Z * [new branch] gh/jacobszwejbka/33/base -> origin/gh/jacobszwejbka/33/base 2022-03-17T19:04:58.5540687Z * [new branch] gh/jacobszwejbka/35/base -> origin/gh/jacobszwejbka/35/base 2022-03-17T19:04:58.5542051Z * [new branch] gh/jacobszwejbka/35/head -> origin/gh/jacobszwejbka/35/head 2022-03-17T19:04:58.5543772Z * [new branch] gh/jacobszwejbka/36/base -> origin/gh/jacobszwejbka/36/base 2022-03-17T19:04:58.5545139Z * [new branch] gh/jacobszwejbka/36/head -> origin/gh/jacobszwejbka/36/head 2022-03-17T19:04:58.5548211Z * [new branch] gh/jacobszwejbka/36/orig -> origin/gh/jacobszwejbka/36/orig 2022-03-17T19:04:58.5548595Z * [new branch] gh/jacobszwejbka/43/base -> origin/gh/jacobszwejbka/43/base 2022-03-17T19:04:58.5550207Z * [new branch] gh/jacobszwejbka/43/head -> origin/gh/jacobszwejbka/43/head 2022-03-17T19:04:58.5551056Z * [new branch] gh/jacobszwejbka/43/orig -> origin/gh/jacobszwejbka/43/orig 2022-03-17T19:04:58.5552958Z * [new branch] gh/jamesr66a/214/head -> origin/gh/jamesr66a/214/head 2022-03-17T19:04:58.5554165Z * [new branch] gh/jamesr66a/214/orig -> origin/gh/jamesr66a/214/orig 2022-03-17T19:04:58.5556026Z * [new branch] gh/jamesr66a/261/base -> origin/gh/jamesr66a/261/base 2022-03-17T19:04:58.5557230Z * [new branch] gh/jamesr66a/261/head -> origin/gh/jamesr66a/261/head 2022-03-17T19:04:58.5558703Z * [new branch] gh/jamesr66a/261/orig -> origin/gh/jamesr66a/261/orig 2022-03-17T19:04:58.5560389Z * [new branch] gh/jamesr66a/270/base -> origin/gh/jamesr66a/270/base 2022-03-17T19:04:58.5561898Z * [new branch] gh/jamesr66a/270/head -> origin/gh/jamesr66a/270/head 2022-03-17T19:04:58.5563610Z * [new branch] gh/jamesr66a/270/orig -> origin/gh/jamesr66a/270/orig 2022-03-17T19:04:58.5565396Z * [new branch] gh/jamesr66a/272/base -> origin/gh/jamesr66a/272/base 2022-03-17T19:04:58.5566893Z * [new branch] gh/jamesr66a/272/head -> origin/gh/jamesr66a/272/head 2022-03-17T19:04:58.5568274Z * [new branch] gh/jamesr66a/272/orig -> origin/gh/jamesr66a/272/orig 2022-03-17T19:04:58.5569961Z * [new branch] gh/jamesr66a/277/base -> origin/gh/jamesr66a/277/base 2022-03-17T19:04:58.5571110Z * [new branch] gh/jamesr66a/277/head -> origin/gh/jamesr66a/277/head 2022-03-17T19:04:58.5572584Z * [new branch] gh/jamesr66a/277/orig -> origin/gh/jamesr66a/277/orig 2022-03-17T19:04:58.5573991Z * [new branch] gh/jamesr66a/278/base -> origin/gh/jamesr66a/278/base 2022-03-17T19:04:58.5575511Z * [new branch] gh/jamesr66a/278/head -> origin/gh/jamesr66a/278/head 2022-03-17T19:04:58.5576955Z * [new branch] gh/jamesr66a/278/orig -> origin/gh/jamesr66a/278/orig 2022-03-17T19:04:58.5578739Z * [new branch] gh/jamesr66a/280/base -> origin/gh/jamesr66a/280/base 2022-03-17T19:04:58.5580271Z * [new branch] gh/jamesr66a/280/head -> origin/gh/jamesr66a/280/head 2022-03-17T19:04:58.5581315Z * [new branch] gh/jamesr66a/280/orig -> origin/gh/jamesr66a/280/orig 2022-03-17T19:04:58.5583199Z * [new branch] gh/jamesr66a/282/base -> origin/gh/jamesr66a/282/base 2022-03-17T19:04:58.5584417Z * [new branch] gh/jamesr66a/282/head -> origin/gh/jamesr66a/282/head 2022-03-17T19:04:58.5585887Z * [new branch] gh/jamesr66a/282/orig -> origin/gh/jamesr66a/282/orig 2022-03-17T19:04:58.5587611Z * [new branch] gh/jamesr66a/288/base -> origin/gh/jamesr66a/288/base 2022-03-17T19:04:58.5589045Z * [new branch] gh/jamesr66a/288/head -> origin/gh/jamesr66a/288/head 2022-03-17T19:04:58.5590266Z * [new branch] gh/jamesr66a/288/orig -> origin/gh/jamesr66a/288/orig 2022-03-17T19:04:58.5592054Z * [new branch] gh/jamesr66a/311/base -> origin/gh/jamesr66a/311/base 2022-03-17T19:04:58.5593482Z * [new branch] gh/jamesr66a/311/head -> origin/gh/jamesr66a/311/head 2022-03-17T19:04:58.5594726Z * [new branch] gh/jamesr66a/311/orig -> origin/gh/jamesr66a/311/orig 2022-03-17T19:04:58.5596531Z * [new branch] gh/jamesr66a/319/base -> origin/gh/jamesr66a/319/base 2022-03-17T19:04:58.5597987Z * [new branch] gh/jamesr66a/319/head -> origin/gh/jamesr66a/319/head 2022-03-17T19:04:58.5599167Z * [new branch] gh/jamesr66a/319/orig -> origin/gh/jamesr66a/319/orig 2022-03-17T19:04:58.5600903Z * [new branch] gh/jamesr66a/32/base -> origin/gh/jamesr66a/32/base 2022-03-17T19:04:58.5602191Z * [new branch] gh/jamesr66a/32/head -> origin/gh/jamesr66a/32/head 2022-03-17T19:04:58.5604425Z * [new branch] gh/jamesr66a/32/orig -> origin/gh/jamesr66a/32/orig 2022-03-17T19:04:58.5606076Z * [new branch] gh/jamesr66a/322/base -> origin/gh/jamesr66a/322/base 2022-03-17T19:04:58.5608216Z * [new branch] gh/jamesr66a/322/head -> origin/gh/jamesr66a/322/head 2022-03-17T19:04:58.5609412Z * [new branch] gh/jamesr66a/322/orig -> origin/gh/jamesr66a/322/orig 2022-03-17T19:04:58.5611249Z * [new branch] gh/jamesr66a/341/base -> origin/gh/jamesr66a/341/base 2022-03-17T19:04:58.5612688Z * [new branch] gh/jamesr66a/341/head -> origin/gh/jamesr66a/341/head 2022-03-17T19:04:58.5613843Z * [new branch] gh/jamesr66a/341/orig -> origin/gh/jamesr66a/341/orig 2022-03-17T19:04:58.5615801Z * [new branch] gh/jamesr66a/369/base -> origin/gh/jamesr66a/369/base 2022-03-17T19:04:58.5617024Z * [new branch] gh/jamesr66a/369/head -> origin/gh/jamesr66a/369/head 2022-03-17T19:04:58.5618457Z * [new branch] gh/jamesr66a/369/orig -> origin/gh/jamesr66a/369/orig 2022-03-17T19:04:58.5620162Z * [new branch] gh/jamesr66a/370/base -> origin/gh/jamesr66a/370/base 2022-03-17T19:04:58.5621681Z * [new branch] gh/jamesr66a/370/head -> origin/gh/jamesr66a/370/head 2022-03-17T19:04:58.5622871Z * [new branch] gh/jamesr66a/370/orig -> origin/gh/jamesr66a/370/orig 2022-03-17T19:04:58.5624413Z * [new branch] gh/jamesr66a/374/base -> origin/gh/jamesr66a/374/base 2022-03-17T19:04:58.5625836Z * [new branch] gh/jamesr66a/374/head -> origin/gh/jamesr66a/374/head 2022-03-17T19:04:58.5627058Z * [new branch] gh/jamesr66a/374/orig -> origin/gh/jamesr66a/374/orig 2022-03-17T19:04:58.5628909Z * [new branch] gh/jamesr66a/377/base -> origin/gh/jamesr66a/377/base 2022-03-17T19:04:58.5630415Z * [new branch] gh/jamesr66a/377/head -> origin/gh/jamesr66a/377/head 2022-03-17T19:04:58.5631642Z * [new branch] gh/jamesr66a/377/orig -> origin/gh/jamesr66a/377/orig 2022-03-17T19:04:58.5633422Z * [new branch] gh/jamesr66a/399/base -> origin/gh/jamesr66a/399/base 2022-03-17T19:04:58.5634827Z * [new branch] gh/jamesr66a/399/head -> origin/gh/jamesr66a/399/head 2022-03-17T19:04:58.5636093Z * [new branch] gh/jamesr66a/399/orig -> origin/gh/jamesr66a/399/orig 2022-03-17T19:04:58.5637643Z * [new branch] gh/jamesr66a/400/base -> origin/gh/jamesr66a/400/base 2022-03-17T19:04:58.5638852Z * [new branch] gh/jamesr66a/400/head -> origin/gh/jamesr66a/400/head 2022-03-17T19:04:58.5640447Z * [new branch] gh/jamesr66a/400/orig -> origin/gh/jamesr66a/400/orig 2022-03-17T19:04:58.5641693Z * [new branch] gh/jamesr66a/401/base -> origin/gh/jamesr66a/401/base 2022-03-17T19:04:58.5643361Z * [new branch] gh/jamesr66a/401/head -> origin/gh/jamesr66a/401/head 2022-03-17T19:04:58.5644455Z * [new branch] gh/jamesr66a/401/orig -> origin/gh/jamesr66a/401/orig 2022-03-17T19:04:58.5646318Z * [new branch] gh/jamesr66a/403/base -> origin/gh/jamesr66a/403/base 2022-03-17T19:04:58.5647760Z * [new branch] gh/jamesr66a/403/head -> origin/gh/jamesr66a/403/head 2022-03-17T19:04:58.5648962Z * [new branch] gh/jamesr66a/403/orig -> origin/gh/jamesr66a/403/orig 2022-03-17T19:04:58.5650821Z * [new branch] gh/jamesr66a/408/base -> origin/gh/jamesr66a/408/base 2022-03-17T19:04:58.5651999Z * [new branch] gh/jamesr66a/408/head -> origin/gh/jamesr66a/408/head 2022-03-17T19:04:58.5653513Z * [new branch] gh/jamesr66a/408/orig -> origin/gh/jamesr66a/408/orig 2022-03-17T19:04:58.5655116Z * [new branch] gh/jamesr66a/418/base -> origin/gh/jamesr66a/418/base 2022-03-17T19:04:58.5656336Z * [new branch] gh/jamesr66a/418/head -> origin/gh/jamesr66a/418/head 2022-03-17T19:04:58.5657787Z * [new branch] gh/jamesr66a/418/orig -> origin/gh/jamesr66a/418/orig 2022-03-17T19:04:58.5659559Z * [new branch] gh/jamesr66a/420/base -> origin/gh/jamesr66a/420/base 2022-03-17T19:04:58.5660790Z * [new branch] gh/jamesr66a/420/head -> origin/gh/jamesr66a/420/head 2022-03-17T19:04:58.5662206Z * [new branch] gh/jamesr66a/420/orig -> origin/gh/jamesr66a/420/orig 2022-03-17T19:04:58.5663913Z * [new branch] gh/jamesr66a/425/base -> origin/gh/jamesr66a/425/base 2022-03-17T19:04:58.5665105Z * [new branch] gh/jamesr66a/425/head -> origin/gh/jamesr66a/425/head 2022-03-17T19:04:58.5666569Z * [new branch] gh/jamesr66a/425/orig -> origin/gh/jamesr66a/425/orig 2022-03-17T19:04:58.5668278Z * [new branch] gh/jamesr66a/429/base -> origin/gh/jamesr66a/429/base 2022-03-17T19:04:58.5669727Z * [new branch] gh/jamesr66a/429/head -> origin/gh/jamesr66a/429/head 2022-03-17T19:04:58.5670880Z * [new branch] gh/jamesr66a/429/orig -> origin/gh/jamesr66a/429/orig 2022-03-17T19:04:58.5672918Z * [new branch] gh/jamesr66a/434/base -> origin/gh/jamesr66a/434/base 2022-03-17T19:04:58.5674315Z * [new branch] gh/jamesr66a/434/head -> origin/gh/jamesr66a/434/head 2022-03-17T19:04:58.5675508Z * [new branch] gh/jamesr66a/434/orig -> origin/gh/jamesr66a/434/orig 2022-03-17T19:04:58.5677299Z * [new branch] gh/jamesr66a/435/base -> origin/gh/jamesr66a/435/base 2022-03-17T19:04:58.5678735Z * [new branch] gh/jamesr66a/435/head -> origin/gh/jamesr66a/435/head 2022-03-17T19:04:58.5680120Z * [new branch] gh/jamesr66a/435/orig -> origin/gh/jamesr66a/435/orig 2022-03-17T19:04:58.5681946Z * [new branch] gh/jamesr66a/436/base -> origin/gh/jamesr66a/436/base 2022-03-17T19:04:58.5683489Z * [new branch] gh/jamesr66a/436/head -> origin/gh/jamesr66a/436/head 2022-03-17T19:04:58.5684594Z * [new branch] gh/jamesr66a/436/orig -> origin/gh/jamesr66a/436/orig 2022-03-17T19:04:58.5686307Z * [new branch] gh/jamesr66a/51/base -> origin/gh/jamesr66a/51/base 2022-03-17T19:04:58.5693972Z * [new branch] gh/jamesr66a/51/head -> origin/gh/jamesr66a/51/head 2022-03-17T19:04:58.5694880Z * [new branch] gh/jamesr66a/51/orig -> origin/gh/jamesr66a/51/orig 2022-03-17T19:04:58.5695568Z * [new branch] gh/janeyx99/4/head -> origin/gh/janeyx99/4/head 2022-03-17T19:04:58.5696093Z * [new branch] gh/janeyx99/5/base -> origin/gh/janeyx99/5/base 2022-03-17T19:04:58.5696646Z * [new branch] gh/janeyx99/5/head -> origin/gh/janeyx99/5/head 2022-03-17T19:04:58.5697166Z * [new branch] gh/janeyx99/6/base -> origin/gh/janeyx99/6/base 2022-03-17T19:04:58.5697666Z * [new branch] gh/janeyx99/6/head -> origin/gh/janeyx99/6/head 2022-03-17T19:04:58.5698833Z * [new branch] gh/jbschlosser/32/base -> origin/gh/jbschlosser/32/base 2022-03-17T19:04:58.5700206Z * [new branch] gh/jbschlosser/32/head -> origin/gh/jbschlosser/32/head 2022-03-17T19:04:58.5701346Z * [new branch] gh/jbschlosser/32/orig -> origin/gh/jbschlosser/32/orig 2022-03-17T19:04:58.5703524Z * [new branch] gh/jbschlosser/37/base -> origin/gh/jbschlosser/37/base 2022-03-17T19:04:58.5704662Z * [new branch] gh/jbschlosser/37/head -> origin/gh/jbschlosser/37/head 2022-03-17T19:04:58.5706175Z * [new branch] gh/jbschlosser/37/orig -> origin/gh/jbschlosser/37/orig 2022-03-17T19:04:58.5707813Z * [new branch] gh/jbschlosser/38/base -> origin/gh/jbschlosser/38/base 2022-03-17T19:04:58.5708974Z * [new branch] gh/jbschlosser/38/head -> origin/gh/jbschlosser/38/head 2022-03-17T19:04:58.5710462Z * [new branch] gh/jbschlosser/38/orig -> origin/gh/jbschlosser/38/orig 2022-03-17T19:04:58.5712317Z * [new branch] gh/jerryzh168/104/base -> origin/gh/jerryzh168/104/base 2022-03-17T19:04:58.5713714Z * [new branch] gh/jerryzh168/104/head -> origin/gh/jerryzh168/104/head 2022-03-17T19:04:58.5715129Z * [new branch] gh/jerryzh168/104/orig -> origin/gh/jerryzh168/104/orig 2022-03-17T19:04:58.5716929Z * [new branch] gh/jerryzh168/106/base -> origin/gh/jerryzh168/106/base 2022-03-17T19:04:58.5718265Z * [new branch] gh/jerryzh168/106/head -> origin/gh/jerryzh168/106/head 2022-03-17T19:04:58.5719583Z * [new branch] gh/jerryzh168/106/orig -> origin/gh/jerryzh168/106/orig 2022-03-17T19:04:58.5721101Z * [new branch] gh/jerryzh168/107/base -> origin/gh/jerryzh168/107/base 2022-03-17T19:04:58.5722845Z * [new branch] gh/jerryzh168/107/head -> origin/gh/jerryzh168/107/head 2022-03-17T19:04:58.5724252Z * [new branch] gh/jerryzh168/107/orig -> origin/gh/jerryzh168/107/orig 2022-03-17T19:04:58.5753243Z * [new branch] gh/jerryzh168/108/base -> origin/gh/jerryzh168/108/base 2022-03-17T19:04:58.5754480Z * [new branch] gh/jerryzh168/108/head -> origin/gh/jerryzh168/108/head 2022-03-17T19:04:58.5755231Z * [new branch] gh/jerryzh168/108/orig -> origin/gh/jerryzh168/108/orig 2022-03-17T19:04:58.5755763Z * [new branch] gh/jerryzh168/111/base -> origin/gh/jerryzh168/111/base 2022-03-17T19:04:58.5756513Z * [new branch] gh/jerryzh168/111/head -> origin/gh/jerryzh168/111/head 2022-03-17T19:04:58.5757131Z * [new branch] gh/jerryzh168/111/orig -> origin/gh/jerryzh168/111/orig 2022-03-17T19:04:58.5757835Z * [new branch] gh/jerryzh168/116/base -> origin/gh/jerryzh168/116/base 2022-03-17T19:04:58.5758479Z * [new branch] gh/jerryzh168/116/head -> origin/gh/jerryzh168/116/head 2022-03-17T19:04:58.5759064Z * [new branch] gh/jerryzh168/116/orig -> origin/gh/jerryzh168/116/orig 2022-03-17T19:04:58.5759764Z * [new branch] gh/jerryzh168/155/base -> origin/gh/jerryzh168/155/base 2022-03-17T19:04:58.5760298Z * [new branch] gh/jerryzh168/155/head -> origin/gh/jerryzh168/155/head 2022-03-17T19:04:58.5761246Z * [new branch] gh/jerryzh168/155/orig -> origin/gh/jerryzh168/155/orig 2022-03-17T19:04:58.5761973Z * [new branch] gh/jerryzh168/159/base -> origin/gh/jerryzh168/159/base 2022-03-17T19:04:58.5762497Z * [new branch] gh/jerryzh168/159/head -> origin/gh/jerryzh168/159/head 2022-03-17T19:04:58.5763021Z * [new branch] gh/jerryzh168/159/orig -> origin/gh/jerryzh168/159/orig 2022-03-17T19:04:58.5763781Z * [new branch] gh/jerryzh168/191/base -> origin/gh/jerryzh168/191/base 2022-03-17T19:04:58.5764330Z * [new branch] gh/jerryzh168/191/head -> origin/gh/jerryzh168/191/head 2022-03-17T19:04:58.5764967Z * [new branch] gh/jerryzh168/191/orig -> origin/gh/jerryzh168/191/orig 2022-03-17T19:04:58.5765685Z * [new branch] gh/jerryzh168/193/base -> origin/gh/jerryzh168/193/base 2022-03-17T19:04:58.5766264Z * [new branch] gh/jerryzh168/193/head -> origin/gh/jerryzh168/193/head 2022-03-17T19:04:58.5767011Z * [new branch] gh/jerryzh168/193/orig -> origin/gh/jerryzh168/193/orig 2022-03-17T19:04:58.5767614Z * [new branch] gh/jerryzh168/208/base -> origin/gh/jerryzh168/208/base 2022-03-17T19:04:58.5768311Z * [new branch] gh/jerryzh168/208/head -> origin/gh/jerryzh168/208/head 2022-03-17T19:04:58.5768960Z * [new branch] gh/jerryzh168/208/orig -> origin/gh/jerryzh168/208/orig 2022-03-17T19:04:58.5769699Z * [new branch] gh/jerryzh168/221/base -> origin/gh/jerryzh168/221/base 2022-03-17T19:04:58.5770327Z * [new branch] gh/jerryzh168/221/head -> origin/gh/jerryzh168/221/head 2022-03-17T19:04:58.5770915Z * [new branch] gh/jerryzh168/221/orig -> origin/gh/jerryzh168/221/orig 2022-03-17T19:04:58.5771646Z * [new branch] gh/jerryzh168/224/base -> origin/gh/jerryzh168/224/base 2022-03-17T19:04:58.5772241Z * [new branch] gh/jerryzh168/224/head -> origin/gh/jerryzh168/224/head 2022-03-17T19:04:58.5772995Z * [new branch] gh/jerryzh168/224/orig -> origin/gh/jerryzh168/224/orig 2022-03-17T19:04:58.5773594Z * [new branch] gh/jerryzh168/236/base -> origin/gh/jerryzh168/236/base 2022-03-17T19:04:58.5774348Z * [new branch] gh/jerryzh168/236/head -> origin/gh/jerryzh168/236/head 2022-03-17T19:04:58.5774966Z * [new branch] gh/jerryzh168/236/orig -> origin/gh/jerryzh168/236/orig 2022-03-17T19:04:58.5776909Z * [new branch] gh/jerryzh168/238/base -> origin/gh/jerryzh168/238/base 2022-03-17T19:04:58.5777725Z * [new branch] gh/jerryzh168/238/head -> origin/gh/jerryzh168/238/head 2022-03-17T19:04:58.5779208Z * [new branch] gh/jerryzh168/238/orig -> origin/gh/jerryzh168/238/orig 2022-03-17T19:04:58.5781138Z * [new branch] gh/jerryzh168/256/base -> origin/gh/jerryzh168/256/base 2022-03-17T19:04:58.5782246Z * [new branch] gh/jerryzh168/256/head -> origin/gh/jerryzh168/256/head 2022-03-17T19:04:58.5783678Z * [new branch] gh/jerryzh168/256/orig -> origin/gh/jerryzh168/256/orig 2022-03-17T19:04:58.5785361Z * [new branch] gh/jerryzh168/269/base -> origin/gh/jerryzh168/269/base 2022-03-17T19:04:58.5786662Z * [new branch] gh/jerryzh168/269/head -> origin/gh/jerryzh168/269/head 2022-03-17T19:04:58.5787963Z * [new branch] gh/jerryzh168/269/orig -> origin/gh/jerryzh168/269/orig 2022-03-17T19:04:58.5789724Z * [new branch] gh/jerryzh168/28/base -> origin/gh/jerryzh168/28/base 2022-03-17T19:04:58.5791111Z * [new branch] gh/jerryzh168/28/head -> origin/gh/jerryzh168/28/head 2022-03-17T19:04:58.5792489Z * [new branch] gh/jerryzh168/28/orig -> origin/gh/jerryzh168/28/orig 2022-03-17T19:04:58.5794178Z * [new branch] gh/jerryzh168/284/base -> origin/gh/jerryzh168/284/base 2022-03-17T19:04:58.5795506Z * [new branch] gh/jerryzh168/284/head -> origin/gh/jerryzh168/284/head 2022-03-17T19:04:58.5796999Z * [new branch] gh/jerryzh168/284/orig -> origin/gh/jerryzh168/284/orig 2022-03-17T19:04:58.5798904Z * [new branch] gh/jerryzh168/29/base -> origin/gh/jerryzh168/29/base 2022-03-17T19:04:58.5800249Z * [new branch] gh/jerryzh168/29/head -> origin/gh/jerryzh168/29/head 2022-03-17T19:04:58.5801646Z * [new branch] gh/jerryzh168/29/orig -> origin/gh/jerryzh168/29/orig 2022-03-17T19:04:58.5803430Z * [new branch] gh/jerryzh168/30/base -> origin/gh/jerryzh168/30/base 2022-03-17T19:04:58.5804821Z * [new branch] gh/jerryzh168/30/head -> origin/gh/jerryzh168/30/head 2022-03-17T19:04:58.5806231Z * [new branch] gh/jerryzh168/30/orig -> origin/gh/jerryzh168/30/orig 2022-03-17T19:04:58.5808003Z * [new branch] gh/jerryzh168/318/base -> origin/gh/jerryzh168/318/base 2022-03-17T19:04:58.5809209Z * [new branch] gh/jerryzh168/318/head -> origin/gh/jerryzh168/318/head 2022-03-17T19:04:58.5810559Z * [new branch] gh/jerryzh168/318/orig -> origin/gh/jerryzh168/318/orig 2022-03-17T19:04:58.5812464Z * [new branch] gh/jerryzh168/323/base -> origin/gh/jerryzh168/323/base 2022-03-17T19:04:58.5813826Z * [new branch] gh/jerryzh168/323/head -> origin/gh/jerryzh168/323/head 2022-03-17T19:04:58.5815100Z * [new branch] gh/jerryzh168/323/orig -> origin/gh/jerryzh168/323/orig 2022-03-17T19:04:58.5816888Z * [new branch] gh/jerryzh168/337/base -> origin/gh/jerryzh168/337/base 2022-03-17T19:04:58.5818280Z * [new branch] gh/jerryzh168/337/head -> origin/gh/jerryzh168/337/head 2022-03-17T19:04:58.5819636Z * [new branch] gh/jerryzh168/337/orig -> origin/gh/jerryzh168/337/orig 2022-03-17T19:04:58.5821417Z * [new branch] gh/jerryzh168/353/base -> origin/gh/jerryzh168/353/base 2022-03-17T19:04:58.5822757Z * [new branch] gh/jerryzh168/353/head -> origin/gh/jerryzh168/353/head 2022-03-17T19:04:58.5824096Z * [new branch] gh/jerryzh168/353/orig -> origin/gh/jerryzh168/353/orig 2022-03-17T19:04:58.5825855Z * [new branch] gh/jerryzh168/411/base -> origin/gh/jerryzh168/411/base 2022-03-17T19:04:58.5827244Z * [new branch] gh/jerryzh168/411/head -> origin/gh/jerryzh168/411/head 2022-03-17T19:04:58.5828525Z * [new branch] gh/jerryzh168/411/orig -> origin/gh/jerryzh168/411/orig 2022-03-17T19:04:58.5830274Z * [new branch] gh/jerryzh168/412/base -> origin/gh/jerryzh168/412/base 2022-03-17T19:04:58.5831652Z * [new branch] gh/jerryzh168/412/head -> origin/gh/jerryzh168/412/head 2022-03-17T19:04:58.5833172Z * [new branch] gh/jerryzh168/412/orig -> origin/gh/jerryzh168/412/orig 2022-03-17T19:04:58.5834852Z * [new branch] gh/jerryzh168/43/base -> origin/gh/jerryzh168/43/base 2022-03-17T19:04:58.5836208Z * [new branch] gh/jerryzh168/43/head -> origin/gh/jerryzh168/43/head 2022-03-17T19:04:58.5837573Z * [new branch] gh/jerryzh168/43/orig -> origin/gh/jerryzh168/43/orig 2022-03-17T19:04:58.5839522Z * [new branch] gh/jerryzh168/464/base -> origin/gh/jerryzh168/464/base 2022-03-17T19:04:58.5840837Z * [new branch] gh/jerryzh168/464/head -> origin/gh/jerryzh168/464/head 2022-03-17T19:04:58.5842293Z * [new branch] gh/jerryzh168/464/orig -> origin/gh/jerryzh168/464/orig 2022-03-17T19:04:58.5844435Z * [new branch] gh/jerryzh168/47/base -> origin/gh/jerryzh168/47/base 2022-03-17T19:04:58.5845741Z * [new branch] gh/jerryzh168/47/head -> origin/gh/jerryzh168/47/head 2022-03-17T19:04:58.5847135Z * [new branch] gh/jerryzh168/47/orig -> origin/gh/jerryzh168/47/orig 2022-03-17T19:04:58.5848885Z * [new branch] gh/jerryzh168/48/base -> origin/gh/jerryzh168/48/base 2022-03-17T19:04:58.5850214Z * [new branch] gh/jerryzh168/48/head -> origin/gh/jerryzh168/48/head 2022-03-17T19:04:58.5851574Z * [new branch] gh/jerryzh168/48/orig -> origin/gh/jerryzh168/48/orig 2022-03-17T19:04:58.5853294Z * [new branch] gh/jerryzh168/49/base -> origin/gh/jerryzh168/49/base 2022-03-17T19:04:58.5854805Z * [new branch] gh/jerryzh168/49/head -> origin/gh/jerryzh168/49/head 2022-03-17T19:04:58.5856184Z * [new branch] gh/jerryzh168/49/orig -> origin/gh/jerryzh168/49/orig 2022-03-17T19:04:58.5857965Z * [new branch] gh/jerryzh168/52/base -> origin/gh/jerryzh168/52/base 2022-03-17T19:04:58.5860825Z * [new branch] gh/jerryzh168/52/head -> origin/gh/jerryzh168/52/head 2022-03-17T19:04:58.5861359Z * [new branch] gh/jerryzh168/52/orig -> origin/gh/jerryzh168/52/orig 2022-03-17T19:04:58.5862302Z * [new branch] gh/jerryzh168/521/base -> origin/gh/jerryzh168/521/base 2022-03-17T19:04:58.5863806Z * [new branch] gh/jerryzh168/521/head -> origin/gh/jerryzh168/521/head 2022-03-17T19:04:58.5865095Z * [new branch] gh/jerryzh168/521/orig -> origin/gh/jerryzh168/521/orig 2022-03-17T19:04:58.5866590Z * [new branch] gh/jerryzh168/53/base -> origin/gh/jerryzh168/53/base 2022-03-17T19:04:58.5868067Z * [new branch] gh/jerryzh168/53/head -> origin/gh/jerryzh168/53/head 2022-03-17T19:04:58.5869382Z * [new branch] gh/jerryzh168/53/orig -> origin/gh/jerryzh168/53/orig 2022-03-17T19:04:58.5871145Z * [new branch] gh/jerryzh168/54/base -> origin/gh/jerryzh168/54/base 2022-03-17T19:04:58.5872480Z * [new branch] gh/jerryzh168/54/head -> origin/gh/jerryzh168/54/head 2022-03-17T19:04:58.5873871Z * [new branch] gh/jerryzh168/54/orig -> origin/gh/jerryzh168/54/orig 2022-03-17T19:04:58.5875648Z * [new branch] gh/jerryzh168/556/base -> origin/gh/jerryzh168/556/base 2022-03-17T19:04:58.5876966Z * [new branch] gh/jerryzh168/556/head -> origin/gh/jerryzh168/556/head 2022-03-17T19:04:58.5878350Z * [new branch] gh/jerryzh168/556/orig -> origin/gh/jerryzh168/556/orig 2022-03-17T19:04:58.5880256Z * [new branch] gh/jerryzh168/575/base -> origin/gh/jerryzh168/575/base 2022-03-17T19:04:58.5881606Z * [new branch] gh/jerryzh168/575/head -> origin/gh/jerryzh168/575/head 2022-03-17T19:04:58.5883232Z * [new branch] gh/jerryzh168/575/orig -> origin/gh/jerryzh168/575/orig 2022-03-17T19:04:58.5884876Z * [new branch] gh/jerryzh168/576/base -> origin/gh/jerryzh168/576/base 2022-03-17T19:04:58.5886181Z * [new branch] gh/jerryzh168/576/head -> origin/gh/jerryzh168/576/head 2022-03-17T19:04:58.5887666Z * [new branch] gh/jerryzh168/576/orig -> origin/gh/jerryzh168/576/orig 2022-03-17T19:04:58.5889326Z * [new branch] gh/jerryzh168/593/base -> origin/gh/jerryzh168/593/base 2022-03-17T19:04:58.5890692Z * [new branch] gh/jerryzh168/593/head -> origin/gh/jerryzh168/593/head 2022-03-17T19:04:58.5892334Z * [new branch] gh/jerryzh168/593/orig -> origin/gh/jerryzh168/593/orig 2022-03-17T19:04:58.5893771Z * [new branch] gh/jerryzh168/594/base -> origin/gh/jerryzh168/594/base 2022-03-17T19:04:58.5895092Z * [new branch] gh/jerryzh168/594/head -> origin/gh/jerryzh168/594/head 2022-03-17T19:04:58.5896418Z * [new branch] gh/jerryzh168/594/orig -> origin/gh/jerryzh168/594/orig 2022-03-17T19:04:58.5898327Z * [new branch] gh/jerryzh168/631/base -> origin/gh/jerryzh168/631/base 2022-03-17T19:04:58.5899883Z * [new branch] gh/jerryzh168/631/head -> origin/gh/jerryzh168/631/head 2022-03-17T19:04:58.5901197Z * [new branch] gh/jerryzh168/631/orig -> origin/gh/jerryzh168/631/orig 2022-03-17T19:04:58.5903219Z * [new branch] gh/jerryzh168/632/base -> origin/gh/jerryzh168/632/base 2022-03-17T19:04:58.5904673Z * [new branch] gh/jerryzh168/632/head -> origin/gh/jerryzh168/632/head 2022-03-17T19:04:58.5906125Z * [new branch] gh/jerryzh168/632/orig -> origin/gh/jerryzh168/632/orig 2022-03-17T19:04:58.5907685Z * [new branch] gh/jerryzh168/633/base -> origin/gh/jerryzh168/633/base 2022-03-17T19:04:58.5909052Z * [new branch] gh/jerryzh168/633/head -> origin/gh/jerryzh168/633/head 2022-03-17T19:04:58.5910157Z * [new branch] gh/jerryzh168/633/orig -> origin/gh/jerryzh168/633/orig 2022-03-17T19:04:58.5912079Z * [new branch] gh/jerryzh168/634/base -> origin/gh/jerryzh168/634/base 2022-03-17T19:04:58.5913494Z * [new branch] gh/jerryzh168/634/head -> origin/gh/jerryzh168/634/head 2022-03-17T19:04:58.5914757Z * [new branch] gh/jerryzh168/634/orig -> origin/gh/jerryzh168/634/orig 2022-03-17T19:04:58.5916817Z * [new branch] gh/jerryzh168/645/base -> origin/gh/jerryzh168/645/base 2022-03-17T19:04:58.5918211Z * [new branch] gh/jerryzh168/645/head -> origin/gh/jerryzh168/645/head 2022-03-17T19:04:58.5919593Z * [new branch] gh/jerryzh168/645/orig -> origin/gh/jerryzh168/645/orig 2022-03-17T19:04:58.5921393Z * [new branch] gh/jerryzh168/692/base -> origin/gh/jerryzh168/692/base 2022-03-17T19:04:58.5923322Z * [new branch] gh/jerryzh168/692/head -> origin/gh/jerryzh168/692/head 2022-03-17T19:04:58.5925413Z * [new branch] gh/jerryzh168/692/orig -> origin/gh/jerryzh168/692/orig 2022-03-17T19:04:58.5927150Z * [new branch] gh/jerryzh168/70/base -> origin/gh/jerryzh168/70/base 2022-03-17T19:04:58.5928963Z * [new branch] gh/jerryzh168/70/head -> origin/gh/jerryzh168/70/head 2022-03-17T19:04:58.5930109Z * [new branch] gh/jerryzh168/70/orig -> origin/gh/jerryzh168/70/orig 2022-03-17T19:04:58.5931815Z * [new branch] gh/jerryzh168/715/base -> origin/gh/jerryzh168/715/base 2022-03-17T19:04:58.5933203Z * [new branch] gh/jerryzh168/715/head -> origin/gh/jerryzh168/715/head 2022-03-17T19:04:58.5934475Z * [new branch] gh/jerryzh168/715/orig -> origin/gh/jerryzh168/715/orig 2022-03-17T19:04:58.5936504Z * [new branch] gh/jerryzh168/717/base -> origin/gh/jerryzh168/717/base 2022-03-17T19:04:58.5937658Z * [new branch] gh/jerryzh168/717/head -> origin/gh/jerryzh168/717/head 2022-03-17T19:04:58.5938913Z * [new branch] gh/jerryzh168/717/orig -> origin/gh/jerryzh168/717/orig 2022-03-17T19:04:58.5940789Z * [new branch] gh/jerryzh168/718/base -> origin/gh/jerryzh168/718/base 2022-03-17T19:04:58.5942235Z * [new branch] gh/jerryzh168/718/head -> origin/gh/jerryzh168/718/head 2022-03-17T19:04:58.5943547Z * [new branch] gh/jerryzh168/718/orig -> origin/gh/jerryzh168/718/orig 2022-03-17T19:04:58.5945365Z * [new branch] gh/jerryzh168/738/base -> origin/gh/jerryzh168/738/base 2022-03-17T19:04:58.5946683Z * [new branch] gh/jerryzh168/738/head -> origin/gh/jerryzh168/738/head 2022-03-17T19:04:58.5948065Z * [new branch] gh/jerryzh168/738/orig -> origin/gh/jerryzh168/738/orig 2022-03-17T19:04:58.5949811Z * [new branch] gh/jerryzh168/742/base -> origin/gh/jerryzh168/742/base 2022-03-17T19:04:58.5950898Z * [new branch] gh/jerryzh168/742/head -> origin/gh/jerryzh168/742/head 2022-03-17T19:04:58.5952284Z * [new branch] gh/jerryzh168/742/orig -> origin/gh/jerryzh168/742/orig 2022-03-17T19:04:58.5954028Z * [new branch] gh/jerryzh168/743/base -> origin/gh/jerryzh168/743/base 2022-03-17T19:04:58.5955430Z * [new branch] gh/jerryzh168/743/head -> origin/gh/jerryzh168/743/head 2022-03-17T19:04:58.5956871Z * [new branch] gh/jerryzh168/743/orig -> origin/gh/jerryzh168/743/orig 2022-03-17T19:04:58.5958840Z * [new branch] gh/jerryzh168/744/base -> origin/gh/jerryzh168/744/base 2022-03-17T19:04:58.5960203Z * [new branch] gh/jerryzh168/744/head -> origin/gh/jerryzh168/744/head 2022-03-17T19:04:58.5961584Z * [new branch] gh/jerryzh168/744/orig -> origin/gh/jerryzh168/744/orig 2022-03-17T19:04:58.5963664Z * [new branch] gh/jerryzh168/745/base -> origin/gh/jerryzh168/745/base 2022-03-17T19:04:58.5965055Z * [new branch] gh/jerryzh168/745/head -> origin/gh/jerryzh168/745/head 2022-03-17T19:04:58.5966472Z * [new branch] gh/jerryzh168/745/orig -> origin/gh/jerryzh168/745/orig 2022-03-17T19:04:58.5968046Z * [new branch] gh/jerryzh168/746/base -> origin/gh/jerryzh168/746/base 2022-03-17T19:04:58.5969202Z * [new branch] gh/jerryzh168/746/head -> origin/gh/jerryzh168/746/head 2022-03-17T19:04:58.5970584Z * [new branch] gh/jerryzh168/746/orig -> origin/gh/jerryzh168/746/orig 2022-03-17T19:04:58.5972407Z * [new branch] gh/jerryzh168/747/base -> origin/gh/jerryzh168/747/base 2022-03-17T19:04:58.5973819Z * [new branch] gh/jerryzh168/747/head -> origin/gh/jerryzh168/747/head 2022-03-17T19:04:58.5975234Z * [new branch] gh/jerryzh168/747/orig -> origin/gh/jerryzh168/747/orig 2022-03-17T19:04:58.5977109Z * [new branch] gh/jerryzh168/748/base -> origin/gh/jerryzh168/748/base 2022-03-17T19:04:58.5978257Z * [new branch] gh/jerryzh168/748/head -> origin/gh/jerryzh168/748/head 2022-03-17T19:04:58.5979711Z * [new branch] gh/jerryzh168/748/orig -> origin/gh/jerryzh168/748/orig 2022-03-17T19:04:58.5981422Z * [new branch] gh/jerryzh168/75/base -> origin/gh/jerryzh168/75/base 2022-03-17T19:04:58.5982810Z * [new branch] gh/jerryzh168/75/head -> origin/gh/jerryzh168/75/head 2022-03-17T19:04:58.5984246Z * [new branch] gh/jerryzh168/75/orig -> origin/gh/jerryzh168/75/orig 2022-03-17T19:04:58.5985982Z * [new branch] gh/jerryzh168/96/base -> origin/gh/jerryzh168/96/base 2022-03-17T19:04:58.5987350Z * [new branch] gh/jerryzh168/96/head -> origin/gh/jerryzh168/96/head 2022-03-17T19:04:58.5988699Z * [new branch] gh/jerryzh168/96/orig -> origin/gh/jerryzh168/96/orig 2022-03-17T19:04:58.5990445Z * [new branch] gh/jianyuh/15/base -> origin/gh/jianyuh/15/base 2022-03-17T19:04:58.5991841Z * [new branch] gh/jianyuh/15/head -> origin/gh/jianyuh/15/head 2022-03-17T19:04:58.5993273Z * [new branch] gh/jianyuh/15/orig -> origin/gh/jianyuh/15/orig 2022-03-17T19:04:58.5995133Z * [new branch] gh/jianyuh/18/base -> origin/gh/jianyuh/18/base 2022-03-17T19:04:58.5996344Z * [new branch] gh/jianyuh/18/head -> origin/gh/jianyuh/18/head 2022-03-17T19:04:58.5997730Z * [new branch] gh/jianyuh/18/orig -> origin/gh/jianyuh/18/orig 2022-03-17T19:04:58.5999489Z * [new branch] gh/jianyuh/28/base -> origin/gh/jianyuh/28/base 2022-03-17T19:04:58.6000860Z * [new branch] gh/jianyuh/28/head -> origin/gh/jianyuh/28/head 2022-03-17T19:04:58.6002324Z * [new branch] gh/jianyuh/28/orig -> origin/gh/jianyuh/28/orig 2022-03-17T19:04:58.6004175Z * [new branch] gh/jianyuh/3/base -> origin/gh/jianyuh/3/base 2022-03-17T19:04:58.6005526Z * [new branch] gh/jianyuh/3/head -> origin/gh/jianyuh/3/head 2022-03-17T19:04:58.6006866Z * [new branch] gh/jianyuh/3/orig -> origin/gh/jianyuh/3/orig 2022-03-17T19:04:58.6008416Z * [new branch] gh/jianyuh/43/base -> origin/gh/jianyuh/43/base 2022-03-17T19:04:58.6009928Z * [new branch] gh/jianyuh/43/head -> origin/gh/jianyuh/43/head 2022-03-17T19:04:58.6011459Z * [new branch] gh/jianyuh/43/orig -> origin/gh/jianyuh/43/orig 2022-03-17T19:04:58.6013258Z * [new branch] gh/jianyuh/45/base -> origin/gh/jianyuh/45/base 2022-03-17T19:04:58.6015147Z * [new branch] gh/jianyuh/45/head -> origin/gh/jianyuh/45/head 2022-03-17T19:04:58.6016406Z * [new branch] gh/jianyuh/45/orig -> origin/gh/jianyuh/45/orig 2022-03-17T19:04:58.6017918Z * [new branch] gh/jianyuh/46/base -> origin/gh/jianyuh/46/base 2022-03-17T19:04:58.6019243Z * [new branch] gh/jianyuh/46/head -> origin/gh/jianyuh/46/head 2022-03-17T19:04:58.6020606Z * [new branch] gh/jianyuh/46/orig -> origin/gh/jianyuh/46/orig 2022-03-17T19:04:58.6022415Z * [new branch] gh/jianyuh/49/base -> origin/gh/jianyuh/49/base 2022-03-17T19:04:58.6023658Z * [new branch] gh/jianyuh/49/head -> origin/gh/jianyuh/49/head 2022-03-17T19:04:58.6025025Z * [new branch] gh/jianyuh/49/orig -> origin/gh/jianyuh/49/orig 2022-03-17T19:04:58.6026809Z * [new branch] gh/jianyuh/51/base -> origin/gh/jianyuh/51/base 2022-03-17T19:04:58.6028032Z * [new branch] gh/jianyuh/51/head -> origin/gh/jianyuh/51/head 2022-03-17T19:04:58.6029398Z * [new branch] gh/jianyuh/51/orig -> origin/gh/jianyuh/51/orig 2022-03-17T19:04:58.6031132Z * [new branch] gh/jianyuh/57/base -> origin/gh/jianyuh/57/base 2022-03-17T19:04:58.6032467Z * [new branch] gh/jianyuh/57/head -> origin/gh/jianyuh/57/head 2022-03-17T19:04:58.6033819Z * [new branch] gh/jianyuh/57/orig -> origin/gh/jianyuh/57/orig 2022-03-17T19:04:58.6035600Z * [new branch] gh/jianyuh/58/base -> origin/gh/jianyuh/58/base 2022-03-17T19:04:58.6037336Z * [new branch] gh/jianyuh/58/head -> origin/gh/jianyuh/58/head 2022-03-17T19:04:58.6038686Z * [new branch] gh/jianyuh/58/orig -> origin/gh/jianyuh/58/orig 2022-03-17T19:04:58.6040714Z * [new branch] gh/jianyuh/59/base -> origin/gh/jianyuh/59/base 2022-03-17T19:04:58.6042103Z * [new branch] gh/jianyuh/59/head -> origin/gh/jianyuh/59/head 2022-03-17T19:04:58.6043628Z * [new branch] gh/jianyuh/59/orig -> origin/gh/jianyuh/59/orig 2022-03-17T19:04:58.6045411Z * [new branch] gh/jianyuh/62/base -> origin/gh/jianyuh/62/base 2022-03-17T19:04:58.6046820Z * [new branch] gh/jianyuh/62/head -> origin/gh/jianyuh/62/head 2022-03-17T19:04:58.6048354Z * [new branch] gh/jianyuh/62/orig -> origin/gh/jianyuh/62/orig 2022-03-17T19:04:58.6050433Z * [new branch] gh/jianyuh/64/base -> origin/gh/jianyuh/64/base 2022-03-17T19:04:58.6051407Z * [new branch] gh/jianyuh/64/head -> origin/gh/jianyuh/64/head 2022-03-17T19:04:58.6052746Z * [new branch] gh/jianyuh/64/orig -> origin/gh/jianyuh/64/orig 2022-03-17T19:04:58.6054540Z * [new branch] gh/jianyuh/67/base -> origin/gh/jianyuh/67/base 2022-03-17T19:04:58.6056004Z * [new branch] gh/jianyuh/67/head -> origin/gh/jianyuh/67/head 2022-03-17T19:04:58.6057437Z * [new branch] gh/jianyuh/68/base -> origin/gh/jianyuh/68/base 2022-03-17T19:04:58.6058766Z * [new branch] gh/jianyuh/68/head -> origin/gh/jianyuh/68/head 2022-03-17T19:04:58.6060273Z * [new branch] gh/jianyuh/69/base -> origin/gh/jianyuh/69/base 2022-03-17T19:04:58.6061501Z * [new branch] gh/jianyuh/69/head -> origin/gh/jianyuh/69/head 2022-03-17T19:04:58.6063230Z * [new branch] gh/jianyuh/7/base -> origin/gh/jianyuh/7/base 2022-03-17T19:04:58.6064518Z * [new branch] gh/jianyuh/7/head -> origin/gh/jianyuh/7/head 2022-03-17T19:04:58.6065766Z * [new branch] gh/jianyuh/7/orig -> origin/gh/jianyuh/7/orig 2022-03-17T19:04:58.6067789Z * [new branch] gh/jiayisuse/2/base -> origin/gh/jiayisuse/2/base 2022-03-17T19:04:58.6069166Z * [new branch] gh/jiayisuse/2/head -> origin/gh/jiayisuse/2/head 2022-03-17T19:04:58.6070545Z * [new branch] gh/jiayisuse/2/orig -> origin/gh/jiayisuse/2/orig 2022-03-17T19:04:58.6072011Z * [new branch] gh/jiayisuse/3/base -> origin/gh/jiayisuse/3/base 2022-03-17T19:04:58.6073342Z * [new branch] gh/jiayisuse/3/head -> origin/gh/jiayisuse/3/head 2022-03-17T19:04:58.6074733Z * [new branch] gh/jiayisuse/3/orig -> origin/gh/jiayisuse/3/orig 2022-03-17T19:04:58.6076672Z * [new branch] gh/jjlilley/84/base -> origin/gh/jjlilley/84/base 2022-03-17T19:04:58.6077967Z * [new branch] gh/jjlilley/84/head -> origin/gh/jjlilley/84/head 2022-03-17T19:04:58.6079400Z * [new branch] gh/jjlilley/84/orig -> origin/gh/jjlilley/84/orig 2022-03-17T19:04:58.6081364Z * [new branch] gh/jjsjann123/1/base -> origin/gh/jjsjann123/1/base 2022-03-17T19:04:58.6082863Z * [new branch] gh/jjsjann123/1/head -> origin/gh/jjsjann123/1/head 2022-03-17T19:04:58.6085087Z * [new branch] gh/jjsjann123/1/orig -> origin/gh/jjsjann123/1/orig 2022-03-17T19:04:58.6086956Z * [new branch] gh/jshen/10/base -> origin/gh/jshen/10/base 2022-03-17T19:04:58.6088283Z * [new branch] gh/jshen/10/head -> origin/gh/jshen/10/head 2022-03-17T19:04:58.6089751Z * [new branch] gh/jshen/11/base -> origin/gh/jshen/11/base 2022-03-17T19:04:58.6091208Z * [new branch] gh/jshen/11/head -> origin/gh/jshen/11/head 2022-03-17T19:04:58.6092560Z * [new branch] gh/jshen/3/base -> origin/gh/jshen/3/base 2022-03-17T19:04:58.6094063Z * [new branch] gh/jshen/3/head -> origin/gh/jshen/3/head 2022-03-17T19:04:58.6095355Z * [new branch] gh/jshen/3/orig -> origin/gh/jshen/3/orig 2022-03-17T19:04:58.6096893Z * [new branch] gh/jshen/7/base -> origin/gh/jshen/7/base 2022-03-17T19:04:58.6098144Z * [new branch] gh/jshen/7/head -> origin/gh/jshen/7/head 2022-03-17T19:04:58.6099474Z * [new branch] gh/jshen/7/orig -> origin/gh/jshen/7/orig 2022-03-17T19:04:58.6101374Z * [new branch] gh/kazhou/9/base -> origin/gh/kazhou/9/base 2022-03-17T19:04:58.6102746Z * [new branch] gh/kazhou/9/head -> origin/gh/kazhou/9/head 2022-03-17T19:04:58.6104807Z * [new branch] gh/kimishpatel/22/base -> origin/gh/kimishpatel/22/base 2022-03-17T19:04:58.6106024Z * [new branch] gh/kimishpatel/22/head -> origin/gh/kimishpatel/22/head 2022-03-17T19:04:58.6107476Z * [new branch] gh/kimishpatel/23/base -> origin/gh/kimishpatel/23/base 2022-03-17T19:04:58.6109355Z * [new branch] gh/kimishpatel/23/head -> origin/gh/kimishpatel/23/head 2022-03-17T19:04:58.6111161Z * [new branch] gh/kimishpatel/44/base -> origin/gh/kimishpatel/44/base 2022-03-17T19:04:58.6112628Z * [new branch] gh/kimishpatel/44/head -> origin/gh/kimishpatel/44/head 2022-03-17T19:04:58.6113904Z * [new branch] gh/kimishpatel/44/orig -> origin/gh/kimishpatel/44/orig 2022-03-17T19:04:58.6115696Z * [new branch] gh/kimishpatel/46/base -> origin/gh/kimishpatel/46/base 2022-03-17T19:04:58.6117129Z * [new branch] gh/kimishpatel/46/head -> origin/gh/kimishpatel/46/head 2022-03-17T19:04:58.6118474Z * [new branch] gh/kimishpatel/46/orig -> origin/gh/kimishpatel/46/orig 2022-03-17T19:04:58.6120269Z * [new branch] gh/kimishpatel/59/base -> origin/gh/kimishpatel/59/base 2022-03-17T19:04:58.6121504Z * [new branch] gh/kimishpatel/59/head -> origin/gh/kimishpatel/59/head 2022-03-17T19:04:58.6123102Z * [new branch] gh/kimishpatel/59/orig -> origin/gh/kimishpatel/59/orig 2022-03-17T19:04:58.6124476Z * [new branch] gh/kimishpatel/60/base -> origin/gh/kimishpatel/60/base 2022-03-17T19:04:58.6125716Z * [new branch] gh/kimishpatel/60/head -> origin/gh/kimishpatel/60/head 2022-03-17T19:04:58.6127494Z * [new branch] gh/kimishpatel/64/base -> origin/gh/kimishpatel/64/base 2022-03-17T19:04:58.6128895Z * [new branch] gh/kimishpatel/64/head -> origin/gh/kimishpatel/64/head 2022-03-17T19:04:58.6130841Z * [new branch] gh/kimishpatel/65/base -> origin/gh/kimishpatel/65/base 2022-03-17T19:04:58.6132284Z * [new branch] gh/kimishpatel/65/head -> origin/gh/kimishpatel/65/head 2022-03-17T19:04:58.6133954Z * [new branch] gh/kimishpatel/66/base -> origin/gh/kimishpatel/66/base 2022-03-17T19:04:58.6135233Z * [new branch] gh/kimishpatel/66/head -> origin/gh/kimishpatel/66/head 2022-03-17T19:04:58.6137156Z * [new branch] gh/kimishpatel/67/base -> origin/gh/kimishpatel/67/base 2022-03-17T19:04:58.6138439Z * [new branch] gh/kimishpatel/67/head -> origin/gh/kimishpatel/67/head 2022-03-17T19:04:58.6140354Z * [new branch] gh/kimishpatel/70/base -> origin/gh/kimishpatel/70/base 2022-03-17T19:04:58.6141849Z * [new branch] gh/kimishpatel/70/head -> origin/gh/kimishpatel/70/head 2022-03-17T19:04:58.6143386Z * [new branch] gh/kimishpatel/70/orig -> origin/gh/kimishpatel/70/orig 2022-03-17T19:04:58.6145248Z * [new branch] gh/krovatkin/3/base -> origin/gh/krovatkin/3/base 2022-03-17T19:04:58.6146710Z * [new branch] gh/krovatkin/3/head -> origin/gh/krovatkin/3/head 2022-03-17T19:04:58.6147998Z * [new branch] gh/krovatkin/3/orig -> origin/gh/krovatkin/3/orig 2022-03-17T19:04:58.6149640Z * [new branch] gh/krovatkin/4/base -> origin/gh/krovatkin/4/base 2022-03-17T19:04:58.6150946Z * [new branch] gh/krovatkin/4/head -> origin/gh/krovatkin/4/head 2022-03-17T19:04:58.6152306Z * [new branch] gh/krovatkin/4/orig -> origin/gh/krovatkin/4/orig 2022-03-17T19:04:58.6153966Z * [new branch] gh/krovatkin/5/base -> origin/gh/krovatkin/5/base 2022-03-17T19:04:58.6155356Z * [new branch] gh/krovatkin/5/head -> origin/gh/krovatkin/5/head 2022-03-17T19:04:58.6156650Z * [new branch] gh/krovatkin/5/orig -> origin/gh/krovatkin/5/orig 2022-03-17T19:04:58.6158123Z * [new branch] gh/krovatkin/6/base -> origin/gh/krovatkin/6/base 2022-03-17T19:04:58.6159693Z * [new branch] gh/krovatkin/6/head -> origin/gh/krovatkin/6/head 2022-03-17T19:04:58.6160776Z * [new branch] gh/krovatkin/6/orig -> origin/gh/krovatkin/6/orig 2022-03-17T19:04:58.6162537Z * [new branch] gh/krovatkin/7/base -> origin/gh/krovatkin/7/base 2022-03-17T19:04:58.6164026Z * [new branch] gh/krovatkin/7/head -> origin/gh/krovatkin/7/head 2022-03-17T19:04:58.6165373Z * [new branch] gh/krovatkin/8/base -> origin/gh/krovatkin/8/base 2022-03-17T19:04:58.6166813Z * [new branch] gh/krovatkin/8/head -> origin/gh/krovatkin/8/head 2022-03-17T19:04:58.6168736Z * [new branch] gh/linux-jedi/1/base -> origin/gh/linux-jedi/1/base 2022-03-17T19:04:58.6170058Z * [new branch] gh/linux-jedi/1/orig -> origin/gh/linux-jedi/1/orig 2022-03-17T19:04:58.6171965Z * [new branch] gh/lishanYang/1/base -> origin/gh/lishanYang/1/base 2022-03-17T19:04:58.6173376Z * [new branch] gh/lishanYang/1/head -> origin/gh/lishanYang/1/head 2022-03-17T19:04:58.6175312Z * [new branch] gh/lishanYang/2/base -> origin/gh/lishanYang/2/base 2022-03-17T19:04:58.6176513Z * [new branch] gh/lishanYang/2/head -> origin/gh/lishanYang/2/head 2022-03-17T19:04:58.6178090Z * [new branch] gh/lishanYang/2/orig -> origin/gh/lishanYang/2/orig 2022-03-17T19:04:58.6179389Z * [new branch] gh/lishanYang/3/base -> origin/gh/lishanYang/3/base 2022-03-17T19:04:58.6180729Z * [new branch] gh/lishanYang/3/head -> origin/gh/lishanYang/3/head 2022-03-17T19:04:58.6182483Z * [new branch] gh/lishanYang/3/orig -> origin/gh/lishanYang/3/orig 2022-03-17T19:04:58.6183974Z * [new branch] gh/lishanYang/4/base -> origin/gh/lishanYang/4/base 2022-03-17T19:04:58.6185401Z * [new branch] gh/lishanYang/4/head -> origin/gh/lishanYang/4/head 2022-03-17T19:04:58.6186709Z * [new branch] gh/lishanYang/4/orig -> origin/gh/lishanYang/4/orig 2022-03-17T19:04:58.6188401Z * [new branch] gh/ljk53/117/base -> origin/gh/ljk53/117/base 2022-03-17T19:04:58.6189882Z * [new branch] gh/ljk53/117/orig -> origin/gh/ljk53/117/orig 2022-03-17T19:04:58.6191847Z * [new branch] gh/ljk53/123/orig -> origin/gh/ljk53/123/orig 2022-03-17T19:04:58.6193707Z * [new branch] gh/ljk53/135/base -> origin/gh/ljk53/135/base 2022-03-17T19:04:58.6195085Z * [new branch] gh/ljk53/135/orig -> origin/gh/ljk53/135/orig 2022-03-17T19:04:58.6196541Z * [new branch] gh/ljk53/136/base -> origin/gh/ljk53/136/base 2022-03-17T19:04:58.6198030Z * [new branch] gh/ljk53/136/orig -> origin/gh/ljk53/136/orig 2022-03-17T19:04:58.6199819Z * [new branch] gh/ljk53/147/base -> origin/gh/ljk53/147/base 2022-03-17T19:04:58.6201198Z * [new branch] gh/ljk53/147/head -> origin/gh/ljk53/147/head 2022-03-17T19:04:58.6202738Z * [new branch] gh/ljk53/147/orig -> origin/gh/ljk53/147/orig 2022-03-17T19:04:58.6204401Z * [new branch] gh/ljk53/19/base -> origin/gh/ljk53/19/base 2022-03-17T19:04:58.6205760Z * [new branch] gh/ljk53/19/orig -> origin/gh/ljk53/19/orig 2022-03-17T19:04:58.6207553Z * [new branch] gh/ljk53/20/orig -> origin/gh/ljk53/20/orig 2022-03-17T19:04:58.6209105Z * [new branch] gh/ljk53/5/base -> origin/gh/ljk53/5/base 2022-03-17T19:04:58.6210929Z * [new branch] gh/ljk53/54/head -> origin/gh/ljk53/54/head 2022-03-17T19:04:58.6212202Z * [new branch] gh/ljk53/54/orig -> origin/gh/ljk53/54/orig 2022-03-17T19:04:58.6213982Z * [new branch] gh/ljk53/58/head -> origin/gh/ljk53/58/head 2022-03-17T19:04:58.6214962Z * [new branch] gh/ljk53/58/orig -> origin/gh/ljk53/58/orig 2022-03-17T19:04:58.6217180Z * [new branch] gh/lw/102/base -> origin/gh/lw/102/base 2022-03-17T19:04:58.6218816Z * [new branch] gh/lw/102/head -> origin/gh/lw/102/head 2022-03-17T19:04:58.6220431Z * [new branch] gh/lw/102/orig -> origin/gh/lw/102/orig 2022-03-17T19:04:58.6222319Z * [new branch] gh/lw/106/base -> origin/gh/lw/106/base 2022-03-17T19:04:58.6224000Z * [new branch] gh/lw/106/head -> origin/gh/lw/106/head 2022-03-17T19:04:58.6225572Z * [new branch] gh/lw/106/orig -> origin/gh/lw/106/orig 2022-03-17T19:04:58.6227329Z * [new branch] gh/lw/117/base -> origin/gh/lw/117/base 2022-03-17T19:04:58.6228615Z * [new branch] gh/lw/117/head -> origin/gh/lw/117/head 2022-03-17T19:04:58.6230005Z * [new branch] gh/lw/117/orig -> origin/gh/lw/117/orig 2022-03-17T19:04:58.6231949Z * [new branch] gh/lw/153/base -> origin/gh/lw/153/base 2022-03-17T19:04:58.6233311Z * [new branch] gh/lw/153/orig -> origin/gh/lw/153/orig 2022-03-17T19:04:58.6235088Z * [new branch] gh/lw/159/base -> origin/gh/lw/159/base 2022-03-17T19:04:58.6236452Z * [new branch] gh/lw/159/head -> origin/gh/lw/159/head 2022-03-17T19:04:58.6237886Z * [new branch] gh/lw/159/orig -> origin/gh/lw/159/orig 2022-03-17T19:04:58.6239679Z * [new branch] gh/lw/191/base -> origin/gh/lw/191/base 2022-03-17T19:04:58.6241059Z * [new branch] gh/lw/191/head -> origin/gh/lw/191/head 2022-03-17T19:04:58.6242605Z * [new branch] gh/lw/191/orig -> origin/gh/lw/191/orig 2022-03-17T19:04:58.6245302Z * [new branch] gh/lw/219/base -> origin/gh/lw/219/base 2022-03-17T19:04:58.6246632Z * [new branch] gh/lw/219/head -> origin/gh/lw/219/head 2022-03-17T19:04:58.6248019Z * [new branch] gh/lw/219/orig -> origin/gh/lw/219/orig 2022-03-17T19:04:58.6249740Z * [new branch] gh/lw/223/base -> origin/gh/lw/223/base 2022-03-17T19:04:58.6251114Z * [new branch] gh/lw/223/head -> origin/gh/lw/223/head 2022-03-17T19:04:58.6252510Z * [new branch] gh/lw/223/orig -> origin/gh/lw/223/orig 2022-03-17T19:04:58.6254183Z * [new branch] gh/lw/225/base -> origin/gh/lw/225/base 2022-03-17T19:04:58.6255623Z * [new branch] gh/lw/225/head -> origin/gh/lw/225/head 2022-03-17T19:04:58.6257019Z * [new branch] gh/lw/225/orig -> origin/gh/lw/225/orig 2022-03-17T19:04:58.6258743Z * [new branch] gh/lw/71/base -> origin/gh/lw/71/base 2022-03-17T19:04:58.6260032Z * [new branch] gh/lw/71/head -> origin/gh/lw/71/head 2022-03-17T19:04:58.6261447Z * [new branch] gh/lw/71/orig -> origin/gh/lw/71/orig 2022-03-17T19:04:58.6263193Z * [new branch] gh/lw/75/base -> origin/gh/lw/75/base 2022-03-17T19:04:58.6264583Z * [new branch] gh/lw/75/head -> origin/gh/lw/75/head 2022-03-17T19:04:58.6265983Z * [new branch] gh/lw/75/orig -> origin/gh/lw/75/orig 2022-03-17T19:04:58.6267668Z * [new branch] gh/lw/78/base -> origin/gh/lw/78/base 2022-03-17T19:04:58.6269140Z * [new branch] gh/lw/78/head -> origin/gh/lw/78/head 2022-03-17T19:04:58.6270640Z * [new branch] gh/lw/78/orig -> origin/gh/lw/78/orig 2022-03-17T19:04:58.6272094Z * [new branch] gh/lw/79/base -> origin/gh/lw/79/base 2022-03-17T19:04:58.6273541Z * [new branch] gh/lw/79/head -> origin/gh/lw/79/head 2022-03-17T19:04:58.6275223Z * [new branch] gh/lw/79/orig -> origin/gh/lw/79/orig 2022-03-17T19:04:58.6276577Z * [new branch] gh/lw/80/base -> origin/gh/lw/80/base 2022-03-17T19:04:58.6278027Z * [new branch] gh/lw/80/head -> origin/gh/lw/80/head 2022-03-17T19:04:58.6279414Z * [new branch] gh/lw/80/orig -> origin/gh/lw/80/orig 2022-03-17T19:04:58.6280905Z * [new branch] gh/lw/81/base -> origin/gh/lw/81/base 2022-03-17T19:04:58.6282547Z * [new branch] gh/lw/81/head -> origin/gh/lw/81/head 2022-03-17T19:04:58.6283944Z * [new branch] gh/lw/81/orig -> origin/gh/lw/81/orig 2022-03-17T19:04:58.6285385Z * [new branch] gh/lw/82/base -> origin/gh/lw/82/base 2022-03-17T19:04:58.6286881Z * [new branch] gh/lw/82/head -> origin/gh/lw/82/head 2022-03-17T19:04:58.6288300Z * [new branch] gh/lw/82/orig -> origin/gh/lw/82/orig 2022-03-17T19:04:58.6290219Z * [new branch] gh/lyoka/5/base -> origin/gh/lyoka/5/base 2022-03-17T19:04:58.6291535Z * [new branch] gh/lyoka/5/head -> origin/gh/lyoka/5/head 2022-03-17T19:04:58.6292782Z * [new branch] gh/lyoka/5/orig -> origin/gh/lyoka/5/orig 2022-03-17T19:04:58.6294704Z * [new branch] gh/makslevental/1/base -> origin/gh/makslevental/1/base 2022-03-17T19:04:58.6295991Z * [new branch] gh/makslevental/1/head -> origin/gh/makslevental/1/head 2022-03-17T19:04:58.6297364Z * [new branch] gh/makslevental/1/orig -> origin/gh/makslevental/1/orig 2022-03-17T19:04:58.6299288Z * [new branch] gh/makslevental/10/base -> origin/gh/makslevental/10/base 2022-03-17T19:04:58.6300850Z * [new branch] gh/makslevental/10/head -> origin/gh/makslevental/10/head 2022-03-17T19:04:58.6302235Z * [new branch] gh/makslevental/10/orig -> origin/gh/makslevental/10/orig 2022-03-17T19:04:58.6303952Z * [new branch] gh/makslevental/11/base -> origin/gh/makslevental/11/base 2022-03-17T19:04:58.6305311Z * [new branch] gh/makslevental/11/head -> origin/gh/makslevental/11/head 2022-03-17T19:04:58.6306659Z * [new branch] gh/makslevental/11/orig -> origin/gh/makslevental/11/orig 2022-03-17T19:04:58.6308420Z * [new branch] gh/makslevental/13/base -> origin/gh/makslevental/13/base 2022-03-17T19:04:58.6309777Z * [new branch] gh/makslevental/13/head -> origin/gh/makslevental/13/head 2022-03-17T19:04:58.6311114Z * [new branch] gh/makslevental/13/orig -> origin/gh/makslevental/13/orig 2022-03-17T19:04:58.6312585Z * [new branch] gh/makslevental/15/base -> origin/gh/makslevental/15/base 2022-03-17T19:04:58.6313939Z * [new branch] gh/makslevental/15/head -> origin/gh/makslevental/15/head 2022-03-17T19:04:58.6315855Z * [new branch] gh/makslevental/15/orig -> origin/gh/makslevental/15/orig 2022-03-17T19:04:58.6317483Z * [new branch] gh/makslevental/18/base -> origin/gh/makslevental/18/base 2022-03-17T19:04:58.6318873Z * [new branch] gh/makslevental/18/head -> origin/gh/makslevental/18/head 2022-03-17T19:04:58.6320260Z * [new branch] gh/makslevental/18/orig -> origin/gh/makslevental/18/orig 2022-03-17T19:04:58.6322441Z * [new branch] gh/makslevental/19/base -> origin/gh/makslevental/19/base 2022-03-17T19:04:58.6323894Z * [new branch] gh/makslevental/19/head -> origin/gh/makslevental/19/head 2022-03-17T19:04:58.6325123Z * [new branch] gh/makslevental/19/orig -> origin/gh/makslevental/19/orig 2022-03-17T19:04:58.6327056Z * [new branch] gh/makslevental/27/base -> origin/gh/makslevental/27/base 2022-03-17T19:04:58.6328593Z * [new branch] gh/makslevental/27/head -> origin/gh/makslevental/27/head 2022-03-17T19:04:58.6329703Z * [new branch] gh/makslevental/27/orig -> origin/gh/makslevental/27/orig 2022-03-17T19:04:58.6331335Z * [new branch] gh/makslevental/28/base -> origin/gh/makslevental/28/base 2022-03-17T19:04:58.6332733Z * [new branch] gh/makslevental/28/head -> origin/gh/makslevental/28/head 2022-03-17T19:04:58.6334297Z * [new branch] gh/makslevental/28/orig -> origin/gh/makslevental/28/orig 2022-03-17T19:04:58.6336216Z * [new branch] gh/makslevental/29/base -> origin/gh/makslevental/29/base 2022-03-17T19:04:58.6337378Z * [new branch] gh/makslevental/29/head -> origin/gh/makslevental/29/head 2022-03-17T19:04:58.6338745Z * [new branch] gh/makslevental/29/orig -> origin/gh/makslevental/29/orig 2022-03-17T19:04:58.6340659Z * [new branch] gh/makslevental/30/base -> origin/gh/makslevental/30/base 2022-03-17T19:04:58.6341942Z * [new branch] gh/makslevental/30/head -> origin/gh/makslevental/30/head 2022-03-17T19:04:58.6343479Z * [new branch] gh/makslevental/30/orig -> origin/gh/makslevental/30/orig 2022-03-17T19:04:58.6344952Z * [new branch] gh/makslevental/31/base -> origin/gh/makslevental/31/base 2022-03-17T19:04:58.6346311Z * [new branch] gh/makslevental/31/head -> origin/gh/makslevental/31/head 2022-03-17T19:04:58.6347637Z * [new branch] gh/makslevental/31/orig -> origin/gh/makslevental/31/orig 2022-03-17T19:04:58.6349480Z * [new branch] gh/makslevental/35/base -> origin/gh/makslevental/35/base 2022-03-17T19:04:58.6350730Z * [new branch] gh/makslevental/35/head -> origin/gh/makslevental/35/head 2022-03-17T19:04:58.6352034Z * [new branch] gh/makslevental/35/orig -> origin/gh/makslevental/35/orig 2022-03-17T19:04:58.6354005Z * [new branch] gh/makslevental/36/base -> origin/gh/makslevental/36/base 2022-03-17T19:04:58.6355219Z * [new branch] gh/makslevental/36/orig -> origin/gh/makslevental/36/orig 2022-03-17T19:04:58.6356884Z * [new branch] gh/makslevental/38/base -> origin/gh/makslevental/38/base 2022-03-17T19:04:58.6358188Z * [new branch] gh/makslevental/38/head -> origin/gh/makslevental/38/head 2022-03-17T19:04:58.6359521Z * [new branch] gh/makslevental/38/orig -> origin/gh/makslevental/38/orig 2022-03-17T19:04:58.6361288Z * [new branch] gh/makslevental/40/base -> origin/gh/makslevental/40/base 2022-03-17T19:04:58.6362737Z * [new branch] gh/makslevental/40/head -> origin/gh/makslevental/40/head 2022-03-17T19:04:58.6364295Z * [new branch] gh/makslevental/40/orig -> origin/gh/makslevental/40/orig 2022-03-17T19:04:58.6365757Z * [new branch] gh/makslevental/8/base -> origin/gh/makslevental/8/base 2022-03-17T19:04:58.6367125Z * [new branch] gh/makslevental/8/head -> origin/gh/makslevental/8/head 2022-03-17T19:04:58.6368531Z * [new branch] gh/makslevental/8/orig -> origin/gh/makslevental/8/orig 2022-03-17T19:04:58.6370067Z * [new branch] gh/makslevental/9/base -> origin/gh/makslevental/9/base 2022-03-17T19:04:58.6371643Z * [new branch] gh/makslevental/9/head -> origin/gh/makslevental/9/head 2022-03-17T19:04:58.6372870Z * [new branch] gh/makslevental/9/orig -> origin/gh/makslevental/9/orig 2022-03-17T19:04:58.6375155Z * [new branch] gh/mal2147/1/base -> origin/gh/mal2147/1/base 2022-03-17T19:04:58.6376969Z * [new branch] gh/mal2147/3/base -> origin/gh/mal2147/3/base 2022-03-17T19:04:58.6378438Z * [new branch] gh/mal2147/4/base -> origin/gh/mal2147/4/base 2022-03-17T19:04:58.6380238Z * [new branch] gh/mal2147/5/base -> origin/gh/mal2147/5/base 2022-03-17T19:04:58.6381843Z * [new branch] gh/mal2147/6/base -> origin/gh/mal2147/6/base 2022-03-17T19:04:58.6383160Z * [new branch] gh/mal2147/7/base -> origin/gh/mal2147/7/base 2022-03-17T19:04:58.6385196Z * [new branch] gh/mal2147/9/base -> origin/gh/mal2147/9/base 2022-03-17T19:04:58.6386760Z * [new branch] gh/malfet/1/base -> origin/gh/malfet/1/base 2022-03-17T19:04:58.6388172Z * [new branch] gh/malfet/1/head -> origin/gh/malfet/1/head 2022-03-17T19:04:58.6389749Z * [new branch] gh/malfet/10/base -> origin/gh/malfet/10/base 2022-03-17T19:04:58.6391061Z * [new branch] gh/malfet/10/orig -> origin/gh/malfet/10/orig 2022-03-17T19:04:58.6393286Z * [new branch] gh/marjanfariborz/4/base -> origin/gh/marjanfariborz/4/base 2022-03-17T19:04:58.6394541Z * [new branch] gh/marjanfariborz/4/head -> origin/gh/marjanfariborz/4/head 2022-03-17T19:04:58.6396036Z * [new branch] gh/marjanfariborz/4/orig -> origin/gh/marjanfariborz/4/orig 2022-03-17T19:04:58.6397648Z * [new branch] gh/marjanfariborz/5/base -> origin/gh/marjanfariborz/5/base 2022-03-17T19:04:58.6398964Z * [new branch] gh/marjanfariborz/5/head -> origin/gh/marjanfariborz/5/head 2022-03-17T19:04:58.6400421Z * [new branch] gh/marjanfariborz/5/orig -> origin/gh/marjanfariborz/5/orig 2022-03-17T19:04:58.6402558Z * [new branch] gh/mcr229/1/base -> origin/gh/mcr229/1/base 2022-03-17T19:04:58.6404889Z * [new branch] gh/mcr229/1/head -> origin/gh/mcr229/1/head 2022-03-17T19:04:58.6405901Z * [new branch] gh/mcr229/1/orig -> origin/gh/mcr229/1/orig 2022-03-17T19:04:58.6407958Z * [new branch] gh/mehtanirav/2/base -> origin/gh/mehtanirav/2/base 2022-03-17T19:04:58.6409229Z * [new branch] gh/mehtanirav/2/head -> origin/gh/mehtanirav/2/head 2022-03-17T19:04:58.6410681Z * [new branch] gh/mehtanirav/2/orig -> origin/gh/mehtanirav/2/orig 2022-03-17T19:04:58.6412693Z * [new branch] gh/mikaylagawarecki/39/base -> origin/gh/mikaylagawarecki/39/base 2022-03-17T19:04:58.6413885Z * [new branch] gh/mikaylagawarecki/39/head -> origin/gh/mikaylagawarecki/39/head 2022-03-17T19:04:58.6415161Z * [new branch] gh/mikaylagawarecki/39/orig -> origin/gh/mikaylagawarecki/39/orig 2022-03-17T19:04:58.6416818Z * [new branch] gh/mikaylagawarecki/40/base -> origin/gh/mikaylagawarecki/40/base 2022-03-17T19:04:58.6418063Z * [new branch] gh/mikaylagawarecki/40/head -> origin/gh/mikaylagawarecki/40/head 2022-03-17T19:04:58.6419715Z * [new branch] gh/mikaylagawarecki/40/orig -> origin/gh/mikaylagawarecki/40/orig 2022-03-17T19:04:58.6421135Z * [new branch] gh/mikaylagawarecki/41/base -> origin/gh/mikaylagawarecki/41/base 2022-03-17T19:04:58.6422531Z * [new branch] gh/mikaylagawarecki/41/head -> origin/gh/mikaylagawarecki/41/head 2022-03-17T19:04:58.6423810Z * [new branch] gh/mikaylagawarecki/41/orig -> origin/gh/mikaylagawarecki/41/orig 2022-03-17T19:04:58.6425454Z * [new branch] gh/mikaylagawarecki/42/base -> origin/gh/mikaylagawarecki/42/base 2022-03-17T19:04:58.6426866Z * [new branch] gh/mikaylagawarecki/42/head -> origin/gh/mikaylagawarecki/42/head 2022-03-17T19:04:58.6428092Z * [new branch] gh/mikaylagawarecki/42/orig -> origin/gh/mikaylagawarecki/42/orig 2022-03-17T19:04:58.6429819Z * [new branch] gh/mikaylagawarecki/43/base -> origin/gh/mikaylagawarecki/43/base 2022-03-17T19:04:58.6431229Z * [new branch] gh/mikaylagawarecki/43/head -> origin/gh/mikaylagawarecki/43/head 2022-03-17T19:04:58.6432564Z * [new branch] gh/mikaylagawarecki/43/orig -> origin/gh/mikaylagawarecki/43/orig 2022-03-17T19:04:58.6434615Z * [new branch] gh/mikaylagawarecki/44/base -> origin/gh/mikaylagawarecki/44/base 2022-03-17T19:04:58.6436144Z * [new branch] gh/mikaylagawarecki/44/head -> origin/gh/mikaylagawarecki/44/head 2022-03-17T19:04:58.6437255Z * [new branch] gh/mikaylagawarecki/44/orig -> origin/gh/mikaylagawarecki/44/orig 2022-03-17T19:04:58.6438911Z * [new branch] gh/mikaylagawarecki/45/base -> origin/gh/mikaylagawarecki/45/base 2022-03-17T19:04:58.6440118Z * [new branch] gh/mikaylagawarecki/45/head -> origin/gh/mikaylagawarecki/45/head 2022-03-17T19:04:58.6441428Z * [new branch] gh/mikaylagawarecki/45/orig -> origin/gh/mikaylagawarecki/45/orig 2022-03-17T19:04:58.6443533Z * [new branch] gh/mikaylagawarecki/6/base -> origin/gh/mikaylagawarecki/6/base 2022-03-17T19:04:58.6444768Z * [new branch] gh/mikaylagawarecki/6/orig -> origin/gh/mikaylagawarecki/6/orig 2022-03-17T19:04:58.6446358Z * [new branch] gh/mikaylagawarecki/7/base -> origin/gh/mikaylagawarecki/7/base 2022-03-17T19:04:58.6447681Z * [new branch] gh/mikaylagawarecki/7/orig -> origin/gh/mikaylagawarecki/7/orig 2022-03-17T19:04:58.6449956Z * [new branch] gh/mikeiovine/28/base -> origin/gh/mikeiovine/28/base 2022-03-17T19:04:58.6451090Z * [new branch] gh/mikeiovine/28/head -> origin/gh/mikeiovine/28/head 2022-03-17T19:04:58.6452275Z * [new branch] gh/mikeiovine/28/orig -> origin/gh/mikeiovine/28/orig 2022-03-17T19:04:58.6454427Z * [new branch] gh/mikeiovine/32/base -> origin/gh/mikeiovine/32/base 2022-03-17T19:04:58.6455734Z * [new branch] gh/mikeiovine/32/head -> origin/gh/mikeiovine/32/head 2022-03-17T19:04:58.6457161Z * [new branch] gh/mikeiovine/32/orig -> origin/gh/mikeiovine/32/orig 2022-03-17T19:04:58.6459127Z * [new branch] gh/mikeiovine/33/base -> origin/gh/mikeiovine/33/base 2022-03-17T19:04:58.6460345Z * [new branch] gh/mikeiovine/33/head -> origin/gh/mikeiovine/33/head 2022-03-17T19:04:58.6461728Z * [new branch] gh/mikeiovine/33/orig -> origin/gh/mikeiovine/33/orig 2022-03-17T19:04:58.6463772Z * [new branch] gh/mikeiovine/36/base -> origin/gh/mikeiovine/36/base 2022-03-17T19:04:58.6464812Z * [new branch] gh/mikeiovine/36/head -> origin/gh/mikeiovine/36/head 2022-03-17T19:04:58.6466243Z * [new branch] gh/mikeiovine/36/orig -> origin/gh/mikeiovine/36/orig 2022-03-17T19:04:58.6468190Z * [new branch] gh/mikeiovine/37/base -> origin/gh/mikeiovine/37/base 2022-03-17T19:04:58.6469513Z * [new branch] gh/mikeiovine/37/head -> origin/gh/mikeiovine/37/head 2022-03-17T19:04:58.6471043Z * [new branch] gh/mikeiovine/37/orig -> origin/gh/mikeiovine/37/orig 2022-03-17T19:04:58.6473054Z * [new branch] gh/mikeiovine/38/base -> origin/gh/mikeiovine/38/base 2022-03-17T19:04:58.6474248Z * [new branch] gh/mikeiovine/38/head -> origin/gh/mikeiovine/38/head 2022-03-17T19:04:58.6475580Z * [new branch] gh/mikeiovine/38/orig -> origin/gh/mikeiovine/38/orig 2022-03-17T19:04:58.6477742Z * [new branch] gh/mingfeima/1/base -> origin/gh/mingfeima/1/base 2022-03-17T19:04:58.6478973Z * [new branch] gh/mingfeima/1/head -> origin/gh/mingfeima/1/head 2022-03-17T19:04:58.6480088Z * [new branch] gh/mingfeima/1/orig -> origin/gh/mingfeima/1/orig 2022-03-17T19:04:58.6481750Z * [new branch] gh/mingfeima/11/base -> origin/gh/mingfeima/11/base 2022-03-17T19:04:58.6483411Z * [new branch] gh/mingfeima/11/head -> origin/gh/mingfeima/11/head 2022-03-17T19:04:58.6484672Z * [new branch] gh/mingfeima/11/orig -> origin/gh/mingfeima/11/orig 2022-03-17T19:04:58.6486519Z * [new branch] gh/mingfeima/13/base -> origin/gh/mingfeima/13/base 2022-03-17T19:04:58.6487757Z * [new branch] gh/mingfeima/13/head -> origin/gh/mingfeima/13/head 2022-03-17T19:04:58.6489423Z * [new branch] gh/mingfeima/13/orig -> origin/gh/mingfeima/13/orig 2022-03-17T19:04:58.6491149Z * [new branch] gh/mingfeima/14/base -> origin/gh/mingfeima/14/base 2022-03-17T19:04:58.6492273Z * [new branch] gh/mingfeima/14/head -> origin/gh/mingfeima/14/head 2022-03-17T19:04:58.6493534Z * [new branch] gh/mingfeima/14/orig -> origin/gh/mingfeima/14/orig 2022-03-17T19:04:58.6494916Z * [new branch] gh/mingfeima/20/base -> origin/gh/mingfeima/20/base 2022-03-17T19:04:58.6496110Z * [new branch] gh/mingfeima/20/head -> origin/gh/mingfeima/20/head 2022-03-17T19:04:58.6497359Z * [new branch] gh/mingfeima/20/orig -> origin/gh/mingfeima/20/orig 2022-03-17T19:04:58.6499176Z * [new branch] gh/mingfeima/22/base -> origin/gh/mingfeima/22/base 2022-03-17T19:04:58.6500431Z * [new branch] gh/mingfeima/22/head -> origin/gh/mingfeima/22/head 2022-03-17T19:04:58.6501720Z * [new branch] gh/mingfeima/22/orig -> origin/gh/mingfeima/22/orig 2022-03-17T19:04:58.6524547Z * [new branch] gh/mingfeima/26/base -> origin/gh/mingfeima/26/base 2022-03-17T19:04:58.6525089Z * [new branch] gh/mingfeima/26/head -> origin/gh/mingfeima/26/head 2022-03-17T19:04:58.6525439Z * [new branch] gh/mingfeima/26/orig -> origin/gh/mingfeima/26/orig 2022-03-17T19:04:58.6525874Z * [new branch] gh/mingfeima/27/base -> origin/gh/mingfeima/27/base 2022-03-17T19:04:58.6526362Z * [new branch] gh/mingfeima/28/base -> origin/gh/mingfeima/28/base 2022-03-17T19:04:58.6526836Z * [new branch] gh/mingfeima/28/head -> origin/gh/mingfeima/28/head 2022-03-17T19:04:58.6527228Z * [new branch] gh/mingfeima/28/orig -> origin/gh/mingfeima/28/orig 2022-03-17T19:04:58.6527652Z * [new branch] gh/mingfeima/29/base -> origin/gh/mingfeima/29/base 2022-03-17T19:04:58.6528016Z * [new branch] gh/mingfeima/29/head -> origin/gh/mingfeima/29/head 2022-03-17T19:04:58.6528345Z * [new branch] gh/mingfeima/29/orig -> origin/gh/mingfeima/29/orig 2022-03-17T19:04:58.6528748Z * [new branch] gh/mingfeima/3/base -> origin/gh/mingfeima/3/base 2022-03-17T19:04:58.6529193Z * [new branch] gh/mingfeima/3/head -> origin/gh/mingfeima/3/head 2022-03-17T19:04:58.6529525Z * [new branch] gh/mingfeima/3/orig -> origin/gh/mingfeima/3/orig 2022-03-17T19:04:58.6529844Z * [new branch] gh/mingfeima/30/base -> origin/gh/mingfeima/30/base 2022-03-17T19:04:58.6530168Z * [new branch] gh/mingfeima/30/head -> origin/gh/mingfeima/30/head 2022-03-17T19:04:58.6530643Z * [new branch] gh/mingfeima/31/base -> origin/gh/mingfeima/31/base 2022-03-17T19:04:58.6531007Z * [new branch] gh/mingfeima/31/head -> origin/gh/mingfeima/31/head 2022-03-17T19:04:58.6531345Z * [new branch] gh/mingfeima/31/orig -> origin/gh/mingfeima/31/orig 2022-03-17T19:04:58.6531741Z * [new branch] gh/mingfeima/32/base -> origin/gh/mingfeima/32/base 2022-03-17T19:04:58.6532316Z * [new branch] gh/mingfeima/32/head -> origin/gh/mingfeima/32/head 2022-03-17T19:04:58.6534532Z * [new branch] gh/mingfeima/32/orig -> origin/gh/mingfeima/32/orig 2022-03-17T19:04:58.6535789Z * [new branch] gh/mingfeima/33/base -> origin/gh/mingfeima/33/base 2022-03-17T19:04:58.6536998Z * [new branch] gh/mingfeima/33/head -> origin/gh/mingfeima/33/head 2022-03-17T19:04:58.6538390Z * [new branch] gh/mingfeima/33/orig -> origin/gh/mingfeima/33/orig 2022-03-17T19:04:58.6539916Z * [new branch] gh/mingfeima/34/base -> origin/gh/mingfeima/34/base 2022-03-17T19:04:58.6541298Z * [new branch] gh/mingfeima/34/head -> origin/gh/mingfeima/34/head 2022-03-17T19:04:58.6542861Z * [new branch] gh/mingfeima/34/orig -> origin/gh/mingfeima/34/orig 2022-03-17T19:04:58.6544358Z * [new branch] gh/mingfeima/35/base -> origin/gh/mingfeima/35/base 2022-03-17T19:04:58.6545647Z * [new branch] gh/mingfeima/35/head -> origin/gh/mingfeima/35/head 2022-03-17T19:04:58.6547018Z * [new branch] gh/mingfeima/35/orig -> origin/gh/mingfeima/35/orig 2022-03-17T19:04:58.6548569Z * [new branch] gh/mingfeima/36/base -> origin/gh/mingfeima/36/base 2022-03-17T19:04:58.6550044Z * [new branch] gh/mingfeima/36/head -> origin/gh/mingfeima/36/head 2022-03-17T19:04:58.6551336Z * [new branch] gh/mingfeima/36/orig -> origin/gh/mingfeima/36/orig 2022-03-17T19:04:58.6553310Z * [new branch] gh/mingfeima/38/base -> origin/gh/mingfeima/38/base 2022-03-17T19:04:58.6554822Z * [new branch] gh/mingfeima/39/base -> origin/gh/mingfeima/39/base 2022-03-17T19:04:58.6556259Z * [new branch] gh/mingfeima/39/head -> origin/gh/mingfeima/39/head 2022-03-17T19:04:58.6557666Z * [new branch] gh/mingfeima/39/orig -> origin/gh/mingfeima/39/orig 2022-03-17T19:04:58.6559557Z * [new branch] gh/mingfeima/4/base -> origin/gh/mingfeima/4/base 2022-03-17T19:04:58.6560694Z * [new branch] gh/mingfeima/4/head -> origin/gh/mingfeima/4/head 2022-03-17T19:04:58.6562228Z * [new branch] gh/mingfeima/4/orig -> origin/gh/mingfeima/4/orig 2022-03-17T19:04:58.6564768Z * [new branch] gh/mingfeima/40/base -> origin/gh/mingfeima/40/base 2022-03-17T19:04:58.6565915Z * [new branch] gh/mingfeima/40/head -> origin/gh/mingfeima/40/head 2022-03-17T19:04:58.6567280Z * [new branch] gh/mingfeima/40/orig -> origin/gh/mingfeima/40/orig 2022-03-17T19:04:58.6568739Z * [new branch] gh/mingfeima/41/base -> origin/gh/mingfeima/41/base 2022-03-17T19:04:58.6570118Z * [new branch] gh/mingfeima/41/head -> origin/gh/mingfeima/41/head 2022-03-17T19:04:58.6571364Z * [new branch] gh/mingfeima/41/orig -> origin/gh/mingfeima/41/orig 2022-03-17T19:04:58.6573233Z * [new branch] gh/mingfeima/48/base -> origin/gh/mingfeima/48/base 2022-03-17T19:04:58.6574426Z * [new branch] gh/mingfeima/48/head -> origin/gh/mingfeima/48/head 2022-03-17T19:04:58.6575695Z * [new branch] gh/mingfeima/48/orig -> origin/gh/mingfeima/48/orig 2022-03-17T19:04:58.6577537Z * [new branch] gh/mingfeima/49/base -> origin/gh/mingfeima/49/base 2022-03-17T19:04:58.6578910Z * [new branch] gh/mingfeima/49/head -> origin/gh/mingfeima/49/head 2022-03-17T19:04:58.6580310Z * [new branch] gh/mingfeima/49/orig -> origin/gh/mingfeima/49/orig 2022-03-17T19:04:58.6582155Z * [new branch] gh/mingfeima/50/base -> origin/gh/mingfeima/50/base 2022-03-17T19:04:58.6583330Z * [new branch] gh/mingfeima/50/head -> origin/gh/mingfeima/50/head 2022-03-17T19:04:58.6584610Z * [new branch] gh/mingfeima/50/orig -> origin/gh/mingfeima/50/orig 2022-03-17T19:04:58.6586500Z * [new branch] gh/mingfeima/51/base -> origin/gh/mingfeima/51/base 2022-03-17T19:04:58.6587798Z * [new branch] gh/mingfeima/51/head -> origin/gh/mingfeima/51/head 2022-03-17T19:04:58.6589195Z * [new branch] gh/mingfeima/51/orig -> origin/gh/mingfeima/51/orig 2022-03-17T19:04:58.6590749Z * [new branch] gh/mingfeima/52/base -> origin/gh/mingfeima/52/base 2022-03-17T19:04:58.6592151Z * [new branch] gh/mingfeima/52/head -> origin/gh/mingfeima/52/head 2022-03-17T19:04:58.6593518Z * [new branch] gh/mingfeima/52/orig -> origin/gh/mingfeima/52/orig 2022-03-17T19:04:58.6595496Z * [new branch] gh/mingfeima/53/base -> origin/gh/mingfeima/53/base 2022-03-17T19:04:58.6596728Z * [new branch] gh/mingfeima/53/head -> origin/gh/mingfeima/53/head 2022-03-17T19:04:58.6597946Z * [new branch] gh/mingfeima/53/orig -> origin/gh/mingfeima/53/orig 2022-03-17T19:04:58.6599683Z * [new branch] gh/mingfeima/54/base -> origin/gh/mingfeima/54/base 2022-03-17T19:04:58.6600989Z * [new branch] gh/mingfeima/54/head -> origin/gh/mingfeima/54/head 2022-03-17T19:04:58.6602476Z * [new branch] gh/mingfeima/54/orig -> origin/gh/mingfeima/54/orig 2022-03-17T19:04:58.6604398Z * [new branch] gh/mingfeima/55/base -> origin/gh/mingfeima/55/base 2022-03-17T19:04:58.6605647Z * [new branch] gh/mingfeima/55/head -> origin/gh/mingfeima/55/head 2022-03-17T19:04:58.6606924Z * [new branch] gh/mingfeima/55/orig -> origin/gh/mingfeima/55/orig 2022-03-17T19:04:58.6608628Z * [new branch] gh/mingfeima/56/base -> origin/gh/mingfeima/56/base 2022-03-17T19:04:58.6610025Z * [new branch] gh/mingfeima/56/head -> origin/gh/mingfeima/56/head 2022-03-17T19:04:58.6611298Z * [new branch] gh/mingfeima/56/orig -> origin/gh/mingfeima/56/orig 2022-03-17T19:04:58.6613054Z * [new branch] gh/mingfeima/57/base -> origin/gh/mingfeima/57/base 2022-03-17T19:04:58.6614366Z * [new branch] gh/mingfeima/57/head -> origin/gh/mingfeima/57/head 2022-03-17T19:04:58.6615759Z * [new branch] gh/mingfeima/57/orig -> origin/gh/mingfeima/57/orig 2022-03-17T19:04:58.6617438Z * [new branch] gh/mingfeima/58/base -> origin/gh/mingfeima/58/base 2022-03-17T19:04:58.6618669Z * [new branch] gh/mingfeima/58/head -> origin/gh/mingfeima/58/head 2022-03-17T19:04:58.6620033Z * [new branch] gh/mingfeima/58/orig -> origin/gh/mingfeima/58/orig 2022-03-17T19:04:58.6621659Z * [new branch] gh/mingfeima/59/base -> origin/gh/mingfeima/59/base 2022-03-17T19:04:58.6622910Z * [new branch] gh/mingfeima/59/head -> origin/gh/mingfeima/59/head 2022-03-17T19:04:58.6624316Z * [new branch] gh/mingfeima/59/orig -> origin/gh/mingfeima/59/orig 2022-03-17T19:04:58.6626153Z * [new branch] gh/mingfeima/60/base -> origin/gh/mingfeima/60/base 2022-03-17T19:04:58.6627400Z * [new branch] gh/mingfeima/60/head -> origin/gh/mingfeima/60/head 2022-03-17T19:04:58.6628689Z * [new branch] gh/mingfeima/60/orig -> origin/gh/mingfeima/60/orig 2022-03-17T19:04:58.6630445Z * [new branch] gh/mingfeima/61/base -> origin/gh/mingfeima/61/base 2022-03-17T19:04:58.6631800Z * [new branch] gh/mingfeima/61/head -> origin/gh/mingfeima/61/head 2022-03-17T19:04:58.6633000Z * [new branch] gh/mingfeima/61/orig -> origin/gh/mingfeima/61/orig 2022-03-17T19:04:58.6634754Z * [new branch] gh/mingfeima/62/base -> origin/gh/mingfeima/62/base 2022-03-17T19:04:58.6636065Z * [new branch] gh/mingfeima/62/head -> origin/gh/mingfeima/62/head 2022-03-17T19:04:58.6637371Z * [new branch] gh/mingfeima/62/orig -> origin/gh/mingfeima/62/orig 2022-03-17T19:04:58.6639241Z * [new branch] gh/mingfeima/63/base -> origin/gh/mingfeima/63/base 2022-03-17T19:04:58.6640393Z * [new branch] gh/mingfeima/63/head -> origin/gh/mingfeima/63/head 2022-03-17T19:04:58.6641619Z * [new branch] gh/mingfeima/63/orig -> origin/gh/mingfeima/63/orig 2022-03-17T19:04:58.6643443Z * [new branch] gh/mingfeima/64/base -> origin/gh/mingfeima/64/base 2022-03-17T19:04:58.6644680Z * [new branch] gh/mingfeima/64/head -> origin/gh/mingfeima/64/head 2022-03-17T19:04:58.6645563Z * [new branch] gh/mingfeima/64/orig -> origin/gh/mingfeima/64/orig 2022-03-17T19:04:58.6648391Z * [new branch] gh/mingfeima/65/base -> origin/gh/mingfeima/65/base 2022-03-17T19:04:58.6648775Z * [new branch] gh/mingfeima/65/head -> origin/gh/mingfeima/65/head 2022-03-17T19:04:58.6650791Z * [new branch] gh/mingfeima/65/orig -> origin/gh/mingfeima/65/orig 2022-03-17T19:04:58.6652405Z * [new branch] gh/mingfeima/8/base -> origin/gh/mingfeima/8/base 2022-03-17T19:04:58.6653649Z * [new branch] gh/mingfeima/8/head -> origin/gh/mingfeima/8/head 2022-03-17T19:04:58.6654905Z * [new branch] gh/mingfeima/8/orig -> origin/gh/mingfeima/8/orig 2022-03-17T19:04:58.6656862Z * [new branch] gh/mingzhe09088/1/base -> origin/gh/mingzhe09088/1/base 2022-03-17T19:04:58.6658140Z * [new branch] gh/mingzhe09088/1/head -> origin/gh/mingzhe09088/1/head 2022-03-17T19:04:58.6659509Z * [new branch] gh/mingzhe09088/1/orig -> origin/gh/mingzhe09088/1/orig 2022-03-17T19:04:58.6661157Z * [new branch] gh/mingzhe09088/12/base -> origin/gh/mingzhe09088/12/base 2022-03-17T19:04:58.6662586Z * [new branch] gh/mingzhe09088/12/head -> origin/gh/mingzhe09088/12/head 2022-03-17T19:04:58.6663927Z * [new branch] gh/mingzhe09088/12/orig -> origin/gh/mingzhe09088/12/orig 2022-03-17T19:04:58.6665599Z * [new branch] gh/mingzhe09088/14/base -> origin/gh/mingzhe09088/14/base 2022-03-17T19:04:58.6666992Z * [new branch] gh/mingzhe09088/14/head -> origin/gh/mingzhe09088/14/head 2022-03-17T19:04:58.6668368Z * [new branch] gh/mingzhe09088/14/orig -> origin/gh/mingzhe09088/14/orig 2022-03-17T19:04:58.6669812Z * [new branch] gh/mingzhe09088/2/base -> origin/gh/mingzhe09088/2/base 2022-03-17T19:04:58.6671234Z * [new branch] gh/mingzhe09088/2/head -> origin/gh/mingzhe09088/2/head 2022-03-17T19:04:58.6672694Z * [new branch] gh/mingzhe09088/2/orig -> origin/gh/mingzhe09088/2/orig 2022-03-17T19:04:58.6674697Z * [new branch] gh/mingzhe09088/3/base -> origin/gh/mingzhe09088/3/base 2022-03-17T19:04:58.6675750Z * [new branch] gh/mingzhe09088/3/head -> origin/gh/mingzhe09088/3/head 2022-03-17T19:04:58.6677231Z * [new branch] gh/mingzhe09088/3/orig -> origin/gh/mingzhe09088/3/orig 2022-03-17T19:04:58.6679097Z * [new branch] gh/mingzhe09088/4/base -> origin/gh/mingzhe09088/4/base 2022-03-17T19:04:58.6680274Z * [new branch] gh/mingzhe09088/4/orig -> origin/gh/mingzhe09088/4/orig 2022-03-17T19:04:58.6682067Z * [new branch] gh/mingzhe09088/5/base -> origin/gh/mingzhe09088/5/base 2022-03-17T19:04:58.6683466Z * [new branch] gh/mingzhe09088/5/head -> origin/gh/mingzhe09088/5/head 2022-03-17T19:04:58.6684783Z * [new branch] gh/mingzhe09088/5/orig -> origin/gh/mingzhe09088/5/orig 2022-03-17T19:04:58.6686630Z * [new branch] gh/mingzhe09088/6/base -> origin/gh/mingzhe09088/6/base 2022-03-17T19:04:58.6687920Z * [new branch] gh/mingzhe09088/6/head -> origin/gh/mingzhe09088/6/head 2022-03-17T19:04:58.6689387Z * [new branch] gh/mingzhe09088/6/orig -> origin/gh/mingzhe09088/6/orig 2022-03-17T19:04:58.6690838Z * [new branch] gh/mingzhe09088/7/base -> origin/gh/mingzhe09088/7/base 2022-03-17T19:04:58.6692008Z * [new branch] gh/mingzhe09088/7/head -> origin/gh/mingzhe09088/7/head 2022-03-17T19:04:58.6693266Z * [new branch] gh/mingzhe09088/7/orig -> origin/gh/mingzhe09088/7/orig 2022-03-17T19:04:58.6695174Z * [new branch] gh/mingzhe09088/8/base -> origin/gh/mingzhe09088/8/base 2022-03-17T19:04:58.6696523Z * [new branch] gh/mingzhe09088/8/head -> origin/gh/mingzhe09088/8/head 2022-03-17T19:04:58.6697854Z * [new branch] gh/mingzhe09088/8/orig -> origin/gh/mingzhe09088/8/orig 2022-03-17T19:04:58.6700331Z * [new branch] gh/mrshenli/103/base -> origin/gh/mrshenli/103/base 2022-03-17T19:04:58.6700752Z * [new branch] gh/mrshenli/103/head -> origin/gh/mrshenli/103/head 2022-03-17T19:04:58.6702654Z * [new branch] gh/mrshenli/103/orig -> origin/gh/mrshenli/103/orig 2022-03-17T19:04:58.6704594Z * [new branch] gh/mrshenli/138/base -> origin/gh/mrshenli/138/base 2022-03-17T19:04:58.6705624Z * [new branch] gh/mrshenli/138/head -> origin/gh/mrshenli/138/head 2022-03-17T19:04:58.6707022Z * [new branch] gh/mrshenli/138/orig -> origin/gh/mrshenli/138/orig 2022-03-17T19:04:58.6708884Z * [new branch] gh/mrshenli/165/base -> origin/gh/mrshenli/165/base 2022-03-17T19:04:58.6710152Z * [new branch] gh/mrshenli/165/head -> origin/gh/mrshenli/165/head 2022-03-17T19:04:58.6711571Z * [new branch] gh/mrshenli/165/orig -> origin/gh/mrshenli/165/orig 2022-03-17T19:04:58.6713114Z * [new branch] gh/mrshenli/171/base -> origin/gh/mrshenli/171/base 2022-03-17T19:04:58.6714548Z * [new branch] gh/mrshenli/171/head -> origin/gh/mrshenli/171/head 2022-03-17T19:04:58.6715734Z * [new branch] gh/mrshenli/171/orig -> origin/gh/mrshenli/171/orig 2022-03-17T19:04:58.6717539Z * [new branch] gh/mrshenli/172/base -> origin/gh/mrshenli/172/base 2022-03-17T19:04:58.6718815Z * [new branch] gh/mrshenli/172/head -> origin/gh/mrshenli/172/head 2022-03-17T19:04:58.6720066Z * [new branch] gh/mrshenli/172/orig -> origin/gh/mrshenli/172/orig 2022-03-17T19:04:58.6721537Z * [new branch] gh/mrshenli/173/base -> origin/gh/mrshenli/173/base 2022-03-17T19:04:58.6723006Z * [new branch] gh/mrshenli/173/head -> origin/gh/mrshenli/173/head 2022-03-17T19:04:58.6725248Z * [new branch] gh/mrshenli/173/orig -> origin/gh/mrshenli/173/orig 2022-03-17T19:04:58.6726742Z * [new branch] gh/mrshenli/174/base -> origin/gh/mrshenli/174/base 2022-03-17T19:04:58.6727767Z * [new branch] gh/mrshenli/174/head -> origin/gh/mrshenli/174/head 2022-03-17T19:04:58.6729076Z * [new branch] gh/mrshenli/174/orig -> origin/gh/mrshenli/174/orig 2022-03-17T19:04:58.6730771Z * [new branch] gh/mrshenli/180/base -> origin/gh/mrshenli/180/base 2022-03-17T19:04:58.6732207Z * [new branch] gh/mrshenli/180/head -> origin/gh/mrshenli/180/head 2022-03-17T19:04:58.6733718Z * [new branch] gh/mrshenli/180/orig -> origin/gh/mrshenli/180/orig 2022-03-17T19:04:58.6735128Z * [new branch] gh/mrshenli/183/base -> origin/gh/mrshenli/183/base 2022-03-17T19:04:58.6736682Z * [new branch] gh/mrshenli/183/head -> origin/gh/mrshenli/183/head 2022-03-17T19:04:58.6737918Z * [new branch] gh/mrshenli/183/orig -> origin/gh/mrshenli/183/orig 2022-03-17T19:04:58.6739715Z * [new branch] gh/mrshenli/186/base -> origin/gh/mrshenli/186/base 2022-03-17T19:04:58.6741026Z * [new branch] gh/mrshenli/186/head -> origin/gh/mrshenli/186/head 2022-03-17T19:04:58.6742280Z * [new branch] gh/mrshenli/186/orig -> origin/gh/mrshenli/186/orig 2022-03-17T19:04:58.6744154Z * [new branch] gh/mrshenli/195/base -> origin/gh/mrshenli/195/base 2022-03-17T19:04:58.6745377Z * [new branch] gh/mrshenli/195/head -> origin/gh/mrshenli/195/head 2022-03-17T19:04:58.6746733Z * [new branch] gh/mrshenli/195/orig -> origin/gh/mrshenli/195/orig 2022-03-17T19:04:58.6748602Z * [new branch] gh/mrshenli/227/base -> origin/gh/mrshenli/227/base 2022-03-17T19:04:58.6749863Z * [new branch] gh/mrshenli/227/head -> origin/gh/mrshenli/227/head 2022-03-17T19:04:58.6751275Z * [new branch] gh/mrshenli/227/orig -> origin/gh/mrshenli/227/orig 2022-03-17T19:04:58.6753262Z * [new branch] gh/mrshenli/240/base -> origin/gh/mrshenli/240/base 2022-03-17T19:04:58.6754463Z * [new branch] gh/mrshenli/240/head -> origin/gh/mrshenli/240/head 2022-03-17T19:04:58.6755749Z * [new branch] gh/mrshenli/240/orig -> origin/gh/mrshenli/240/orig 2022-03-17T19:04:58.6757564Z * [new branch] gh/mrshenli/257/base -> origin/gh/mrshenli/257/base 2022-03-17T19:04:58.6758883Z * [new branch] gh/mrshenli/257/head -> origin/gh/mrshenli/257/head 2022-03-17T19:04:58.6760187Z * [new branch] gh/mrshenli/257/orig -> origin/gh/mrshenli/257/orig 2022-03-17T19:04:58.6762121Z * [new branch] gh/mrshenli/288/base -> origin/gh/mrshenli/288/base 2022-03-17T19:04:58.6763565Z * [new branch] gh/mrshenli/288/head -> origin/gh/mrshenli/288/head 2022-03-17T19:04:58.6764860Z * [new branch] gh/mrshenli/288/orig -> origin/gh/mrshenli/288/orig 2022-03-17T19:04:58.6766895Z * [new branch] gh/mrshenli/302/base -> origin/gh/mrshenli/302/base 2022-03-17T19:04:58.6768136Z * [new branch] gh/mrshenli/302/head -> origin/gh/mrshenli/302/head 2022-03-17T19:04:58.6769511Z * [new branch] gh/mrshenli/302/orig -> origin/gh/mrshenli/302/orig 2022-03-17T19:04:58.6771271Z * [new branch] gh/mrshenli/306/base -> origin/gh/mrshenli/306/base 2022-03-17T19:04:58.6772553Z * [new branch] gh/mrshenli/306/head -> origin/gh/mrshenli/306/head 2022-03-17T19:04:58.6773801Z * [new branch] gh/mrshenli/306/orig -> origin/gh/mrshenli/306/orig 2022-03-17T19:04:58.6775790Z * [new branch] gh/mrshenli/307/base -> origin/gh/mrshenli/307/base 2022-03-17T19:04:58.6777388Z * [new branch] gh/mrshenli/307/head -> origin/gh/mrshenli/307/head 2022-03-17T19:04:58.6778709Z * [new branch] gh/mrshenli/307/orig -> origin/gh/mrshenli/307/orig 2022-03-17T19:04:58.6780565Z * [new branch] gh/mrshenli/309/base -> origin/gh/mrshenli/309/base 2022-03-17T19:04:58.6781768Z * [new branch] gh/mrshenli/309/head -> origin/gh/mrshenli/309/head 2022-03-17T19:04:58.6783276Z * [new branch] gh/mrshenli/309/orig -> origin/gh/mrshenli/309/orig 2022-03-17T19:04:58.6784804Z * [new branch] gh/mrshenli/310/base -> origin/gh/mrshenli/310/base 2022-03-17T19:04:58.6786158Z * [new branch] gh/mrshenli/310/head -> origin/gh/mrshenli/310/head 2022-03-17T19:04:58.6787441Z * [new branch] gh/mrshenli/310/orig -> origin/gh/mrshenli/310/orig 2022-03-17T19:04:58.6789154Z * [new branch] gh/mrshenli/319/base -> origin/gh/mrshenli/319/base 2022-03-17T19:04:58.6789993Z * [new branch] gh/mrshenli/319/head -> origin/gh/mrshenli/319/head 2022-03-17T19:04:58.6791733Z * [new branch] gh/mrshenli/319/orig -> origin/gh/mrshenli/319/orig 2022-03-17T19:04:58.6793474Z * [new branch] gh/mrshenli/320/base -> origin/gh/mrshenli/320/base 2022-03-17T19:04:58.6794716Z * [new branch] gh/mrshenli/320/head -> origin/gh/mrshenli/320/head 2022-03-17T19:04:58.6795987Z * [new branch] gh/mrshenli/320/orig -> origin/gh/mrshenli/320/orig 2022-03-17T19:04:58.6797702Z * [new branch] gh/mrshenli/322/base -> origin/gh/mrshenli/322/base 2022-03-17T19:04:58.6798976Z * [new branch] gh/mrshenli/322/head -> origin/gh/mrshenli/322/head 2022-03-17T19:04:58.6800323Z * [new branch] gh/mrshenli/322/orig -> origin/gh/mrshenli/322/orig 2022-03-17T19:04:58.6802366Z * [new branch] gh/mrshenli/323/base -> origin/gh/mrshenli/323/base 2022-03-17T19:04:58.6803881Z * [new branch] gh/mrshenli/323/head -> origin/gh/mrshenli/323/head 2022-03-17T19:04:58.6805310Z * [new branch] gh/mrshenli/323/orig -> origin/gh/mrshenli/323/orig 2022-03-17T19:04:58.6806998Z * [new branch] gh/mrshenli/324/base -> origin/gh/mrshenli/324/base 2022-03-17T19:04:58.6808237Z * [new branch] gh/mrshenli/324/head -> origin/gh/mrshenli/324/head 2022-03-17T19:04:58.6809575Z * [new branch] gh/mrshenli/324/orig -> origin/gh/mrshenli/324/orig 2022-03-17T19:04:58.6811359Z * [new branch] gh/mrshenli/326/base -> origin/gh/mrshenli/326/base 2022-03-17T19:04:58.6812569Z * [new branch] gh/mrshenli/326/head -> origin/gh/mrshenli/326/head 2022-03-17T19:04:58.6813882Z * [new branch] gh/mrshenli/326/orig -> origin/gh/mrshenli/326/orig 2022-03-17T19:04:58.6815736Z * [new branch] gh/mrshenli/328/base -> origin/gh/mrshenli/328/base 2022-03-17T19:04:58.6816924Z * [new branch] gh/mrshenli/328/head -> origin/gh/mrshenli/328/head 2022-03-17T19:04:58.6818340Z * [new branch] gh/mrshenli/328/orig -> origin/gh/mrshenli/328/orig 2022-03-17T19:04:58.6820432Z * [new branch] gh/mrshenli/329/base -> origin/gh/mrshenli/329/base 2022-03-17T19:04:58.6821622Z * [new branch] gh/mrshenli/329/head -> origin/gh/mrshenli/329/head 2022-03-17T19:04:58.6822932Z * [new branch] gh/mrshenli/329/orig -> origin/gh/mrshenli/329/orig 2022-03-17T19:04:58.6824702Z * [new branch] gh/mrshenli/334/base -> origin/gh/mrshenli/334/base 2022-03-17T19:04:58.6826071Z * [new branch] gh/mrshenli/334/head -> origin/gh/mrshenli/334/head 2022-03-17T19:04:58.6827414Z * [new branch] gh/mrshenli/334/orig -> origin/gh/mrshenli/334/orig 2022-03-17T19:04:58.6829336Z * [new branch] gh/mrshenli/335/base -> origin/gh/mrshenli/335/base 2022-03-17T19:04:58.6830602Z * [new branch] gh/mrshenli/335/head -> origin/gh/mrshenli/335/head 2022-03-17T19:04:58.6831923Z * [new branch] gh/mrshenli/335/orig -> origin/gh/mrshenli/335/orig 2022-03-17T19:04:58.6833487Z * [new branch] gh/mrshenli/336/base -> origin/gh/mrshenli/336/base 2022-03-17T19:04:58.6834858Z * [new branch] gh/mrshenli/336/head -> origin/gh/mrshenli/336/head 2022-03-17T19:04:58.6836189Z * [new branch] gh/mrshenli/336/orig -> origin/gh/mrshenli/336/orig 2022-03-17T19:04:58.6837951Z * [new branch] gh/mrshenli/38/base -> origin/gh/mrshenli/38/base 2022-03-17T19:04:58.6839296Z * [new branch] gh/mrshenli/38/head -> origin/gh/mrshenli/38/head 2022-03-17T19:04:58.6840639Z * [new branch] gh/mrshenli/38/orig -> origin/gh/mrshenli/38/orig 2022-03-17T19:04:58.6842542Z * [new branch] gh/mrshenli/93/base -> origin/gh/mrshenli/93/base 2022-03-17T19:04:58.6844114Z * [new branch] gh/mrshenli/93/head -> origin/gh/mrshenli/93/head 2022-03-17T19:04:58.6845392Z * [new branch] gh/mrshenli/93/orig -> origin/gh/mrshenli/93/orig 2022-03-17T19:04:58.6847128Z * [new branch] gh/mrshenli/95/head -> origin/gh/mrshenli/95/head 2022-03-17T19:04:58.6848408Z * [new branch] gh/mrshenli/95/orig -> origin/gh/mrshenli/95/orig 2022-03-17T19:04:58.6850362Z * [new branch] gh/mrzzd/11/base -> origin/gh/mrzzd/11/base 2022-03-17T19:04:58.6851598Z * [new branch] gh/mrzzd/11/head -> origin/gh/mrzzd/11/head 2022-03-17T19:04:58.6852910Z * [new branch] gh/mrzzd/11/orig -> origin/gh/mrzzd/11/orig 2022-03-17T19:04:58.6854916Z * [new branch] gh/msaroufim/5/base -> origin/gh/msaroufim/5/base 2022-03-17T19:04:58.6856191Z * [new branch] gh/msaroufim/5/head -> origin/gh/msaroufim/5/head 2022-03-17T19:04:58.6857391Z * [new branch] gh/msaroufim/5/orig -> origin/gh/msaroufim/5/orig 2022-03-17T19:04:58.6859556Z * [new branch] gh/mvsampath/1/base -> origin/gh/mvsampath/1/base 2022-03-17T19:04:58.6860771Z * [new branch] gh/mvsampath/1/head -> origin/gh/mvsampath/1/head 2022-03-17T19:04:58.6862139Z * [new branch] gh/mvsampath/1/orig -> origin/gh/mvsampath/1/orig 2022-03-17T19:04:58.6864083Z * [new branch] gh/nairbv/19/base -> origin/gh/nairbv/19/base 2022-03-17T19:04:58.6865401Z * [new branch] gh/nairbv/19/head -> origin/gh/nairbv/19/head 2022-03-17T19:04:58.6866728Z * [new branch] gh/nairbv/19/orig -> origin/gh/nairbv/19/orig 2022-03-17T19:04:58.6868401Z * [new branch] gh/nairbv/2/head -> origin/gh/nairbv/2/head 2022-03-17T19:04:58.6869726Z * [new branch] gh/nairbv/2/orig -> origin/gh/nairbv/2/orig 2022-03-17T19:04:58.6871505Z * [new branch] gh/nairbv/24/base -> origin/gh/nairbv/24/base 2022-03-17T19:04:58.6872819Z * [new branch] gh/nairbv/24/head -> origin/gh/nairbv/24/head 2022-03-17T19:04:58.6874219Z * [new branch] gh/nairbv/24/orig -> origin/gh/nairbv/24/orig 2022-03-17T19:04:58.6875764Z * [new branch] gh/nairbv/25/base -> origin/gh/nairbv/25/base 2022-03-17T19:04:58.6877009Z * [new branch] gh/nairbv/25/head -> origin/gh/nairbv/25/head 2022-03-17T19:04:58.6878373Z * [new branch] gh/nairbv/25/orig -> origin/gh/nairbv/25/orig 2022-03-17T19:04:58.6880095Z * [new branch] gh/nairbv/3/base -> origin/gh/nairbv/3/base 2022-03-17T19:04:58.6881376Z * [new branch] gh/nairbv/3/orig -> origin/gh/nairbv/3/orig 2022-03-17T19:04:58.6884641Z * [new branch] gh/narang99/3/base -> origin/gh/narang99/3/base 2022-03-17T19:04:58.6885872Z * [new branch] gh/narang99/3/head -> origin/gh/narang99/3/head 2022-03-17T19:04:58.6887260Z * [new branch] gh/narang99/3/orig -> origin/gh/narang99/3/orig 2022-03-17T19:04:58.6889010Z * [new branch] gh/narang99/4/base -> origin/gh/narang99/4/base 2022-03-17T19:04:58.6890301Z * [new branch] gh/narang99/4/head -> origin/gh/narang99/4/head 2022-03-17T19:04:58.6891659Z * [new branch] gh/narang99/4/orig -> origin/gh/narang99/4/orig 2022-03-17T19:04:58.6893258Z * [new branch] gh/narang99/5/base -> origin/gh/narang99/5/base 2022-03-17T19:04:58.6894470Z * [new branch] gh/narang99/5/head -> origin/gh/narang99/5/head 2022-03-17T19:04:58.6895909Z * [new branch] gh/narang99/5/orig -> origin/gh/narang99/5/orig 2022-03-17T19:04:58.6897464Z * [new branch] gh/navahgar/28/base -> origin/gh/navahgar/28/base 2022-03-17T19:04:58.6898885Z * [new branch] gh/navahgar/28/head -> origin/gh/navahgar/28/head 2022-03-17T19:04:58.6900264Z * [new branch] gh/navahgar/28/orig -> origin/gh/navahgar/28/orig 2022-03-17T19:04:58.6902039Z * [new branch] gh/navahgar/29/base -> origin/gh/navahgar/29/base 2022-03-17T19:04:58.6903372Z * [new branch] gh/navahgar/29/head -> origin/gh/navahgar/29/head 2022-03-17T19:04:58.6904657Z * [new branch] gh/navahgar/29/orig -> origin/gh/navahgar/29/orig 2022-03-17T19:04:58.6906388Z * [new branch] gh/navahgar/7/base -> origin/gh/navahgar/7/base 2022-03-17T19:04:58.6907742Z * [new branch] gh/navahgar/7/head -> origin/gh/navahgar/7/head 2022-03-17T19:04:58.6909063Z * [new branch] gh/navahgar/7/orig -> origin/gh/navahgar/7/orig 2022-03-17T19:04:58.6910559Z * [new branch] gh/navahgar/8/base -> origin/gh/navahgar/8/base 2022-03-17T19:04:58.6911849Z * [new branch] gh/navahgar/8/head -> origin/gh/navahgar/8/head 2022-03-17T19:04:58.6913311Z * [new branch] gh/navahgar/8/orig -> origin/gh/navahgar/8/orig 2022-03-17T19:04:58.6914733Z * [new branch] gh/navahgar/9/base -> origin/gh/navahgar/9/base 2022-03-17T19:04:58.6916010Z * [new branch] gh/navahgar/9/head -> origin/gh/navahgar/9/head 2022-03-17T19:04:58.6917353Z * [new branch] gh/navahgar/9/orig -> origin/gh/navahgar/9/orig 2022-03-17T19:04:58.6919196Z * [new branch] gh/ngimel/2/base -> origin/gh/ngimel/2/base 2022-03-17T19:04:58.6921209Z * [new branch] gh/nikithamalgifb/2/base -> origin/gh/nikithamalgifb/2/base 2022-03-17T19:04:58.6922757Z * [new branch] gh/nikithamalgifb/2/head -> origin/gh/nikithamalgifb/2/head 2022-03-17T19:04:58.6924233Z * [new branch] gh/nikithamalgifb/2/orig -> origin/gh/nikithamalgifb/2/orig 2022-03-17T19:04:58.6925936Z * [new branch] gh/nikithamalgifb/21/base -> origin/gh/nikithamalgifb/21/base 2022-03-17T19:04:58.6927303Z * [new branch] gh/nikithamalgifb/21/head -> origin/gh/nikithamalgifb/21/head 2022-03-17T19:04:58.6928701Z * [new branch] gh/nikithamalgifb/21/orig -> origin/gh/nikithamalgifb/21/orig 2022-03-17T19:04:58.6930418Z * [new branch] gh/nikithamalgifb/25/base -> origin/gh/nikithamalgifb/25/base 2022-03-17T19:04:58.6931675Z * [new branch] gh/nikithamalgifb/25/head -> origin/gh/nikithamalgifb/25/head 2022-03-17T19:04:58.6932922Z * [new branch] gh/nikithamalgifb/25/orig -> origin/gh/nikithamalgifb/25/orig 2022-03-17T19:04:58.6934678Z * [new branch] gh/nikithamalgifb/27/base -> origin/gh/nikithamalgifb/27/base 2022-03-17T19:04:58.6936033Z * [new branch] gh/nikithamalgifb/27/head -> origin/gh/nikithamalgifb/27/head 2022-03-17T19:04:58.6937291Z * [new branch] gh/nikithamalgifb/27/orig -> origin/gh/nikithamalgifb/27/orig 2022-03-17T19:04:58.6942148Z * [new branch] gh/nikithamalgifb/29/base -> origin/gh/nikithamalgifb/29/base 2022-03-17T19:04:58.6942454Z * [new branch] gh/nikithamalgifb/29/head -> origin/gh/nikithamalgifb/29/head 2022-03-17T19:04:58.6942755Z * [new branch] gh/nikithamalgifb/29/orig -> origin/gh/nikithamalgifb/29/orig 2022-03-17T19:04:58.6943319Z * [new branch] gh/nikithamalgifb/30/base -> origin/gh/nikithamalgifb/30/base 2022-03-17T19:04:58.6945567Z * [new branch] gh/nikithamalgifb/30/head -> origin/gh/nikithamalgifb/30/head 2022-03-17T19:04:58.6946380Z * [new branch] gh/nikithamalgifb/30/orig -> origin/gh/nikithamalgifb/30/orig 2022-03-17T19:04:58.6947843Z * [new branch] gh/nikithamalgifb/31/base -> origin/gh/nikithamalgifb/31/base 2022-03-17T19:04:58.6949117Z * [new branch] gh/nikithamalgifb/31/head -> origin/gh/nikithamalgifb/31/head 2022-03-17T19:04:58.6950458Z * [new branch] gh/nikithamalgifb/31/orig -> origin/gh/nikithamalgifb/31/orig 2022-03-17T19:04:58.6952521Z * [new branch] gh/nikithamalgifb/32/base -> origin/gh/nikithamalgifb/32/base 2022-03-17T19:04:58.6953977Z * [new branch] gh/nikithamalgifb/32/head -> origin/gh/nikithamalgifb/32/head 2022-03-17T19:04:58.6954788Z * [new branch] gh/nikithamalgifb/32/orig -> origin/gh/nikithamalgifb/32/orig 2022-03-17T19:04:58.6956837Z * [new branch] gh/nikithamalgifb/41/base -> origin/gh/nikithamalgifb/41/base 2022-03-17T19:04:58.6957856Z * [new branch] gh/nikithamalgifb/41/head -> origin/gh/nikithamalgifb/41/head 2022-03-17T19:04:58.6959216Z * [new branch] gh/nikithamalgifb/41/orig -> origin/gh/nikithamalgifb/41/orig 2022-03-17T19:04:58.6961161Z * [new branch] gh/nikithamalgifb/43/base -> origin/gh/nikithamalgifb/43/base 2022-03-17T19:04:58.6962389Z * [new branch] gh/nikithamalgifb/43/head -> origin/gh/nikithamalgifb/43/head 2022-03-17T19:04:58.6963797Z * [new branch] gh/nikithamalgifb/43/orig -> origin/gh/nikithamalgifb/43/orig 2022-03-17T19:04:58.6965881Z * [new branch] gh/nikithamalgifb/44/base -> origin/gh/nikithamalgifb/44/base 2022-03-17T19:04:58.6966797Z * [new branch] gh/nikithamalgifb/44/head -> origin/gh/nikithamalgifb/44/head 2022-03-17T19:04:58.6968073Z * [new branch] gh/nikithamalgifb/44/orig -> origin/gh/nikithamalgifb/44/orig 2022-03-17T19:04:58.6969992Z * [new branch] gh/nikithamalgifb/46/base -> origin/gh/nikithamalgifb/46/base 2022-03-17T19:04:58.6971101Z * [new branch] gh/nikithamalgifb/46/head -> origin/gh/nikithamalgifb/46/head 2022-03-17T19:04:58.6972368Z * [new branch] gh/nikithamalgifb/46/orig -> origin/gh/nikithamalgifb/46/orig 2022-03-17T19:04:58.6974086Z * [new branch] gh/nikithamalgifb/47/base -> origin/gh/nikithamalgifb/47/base 2022-03-17T19:04:58.6975362Z * [new branch] gh/nikithamalgifb/47/head -> origin/gh/nikithamalgifb/47/head 2022-03-17T19:04:58.6976684Z * [new branch] gh/nikithamalgifb/47/orig -> origin/gh/nikithamalgifb/47/orig 2022-03-17T19:04:58.6978720Z * [new branch] gh/nikithamalgifb/5/base -> origin/gh/nikithamalgifb/5/base 2022-03-17T19:04:58.6979877Z * [new branch] gh/nikithamalgifb/5/head -> origin/gh/nikithamalgifb/5/head 2022-03-17T19:04:58.6981198Z * [new branch] gh/nikithamalgifb/5/orig -> origin/gh/nikithamalgifb/5/orig 2022-03-17T19:04:58.6983082Z * [new branch] gh/nikithamalgifb/50/base -> origin/gh/nikithamalgifb/50/base 2022-03-17T19:04:58.6984194Z * [new branch] gh/nikithamalgifb/50/head -> origin/gh/nikithamalgifb/50/head 2022-03-17T19:04:58.6985518Z * [new branch] gh/nikithamalgifb/50/orig -> origin/gh/nikithamalgifb/50/orig 2022-03-17T19:04:58.6987393Z * [new branch] gh/nikithamalgifb/9/base -> origin/gh/nikithamalgifb/9/base 2022-03-17T19:04:58.6988572Z * [new branch] gh/nikithamalgifb/9/head -> origin/gh/nikithamalgifb/9/head 2022-03-17T19:04:58.6989923Z * [new branch] gh/nikithamalgifb/9/orig -> origin/gh/nikithamalgifb/9/orig 2022-03-17T19:04:58.6991871Z * [new branch] gh/nivekt/34/base -> origin/gh/nivekt/34/base 2022-03-17T19:04:58.6993135Z * [new branch] gh/nivekt/34/head -> origin/gh/nivekt/34/head 2022-03-17T19:04:58.6994441Z * [new branch] gh/nivekt/34/orig -> origin/gh/nivekt/34/orig 2022-03-17T19:04:58.6996239Z * [new branch] gh/nivekt/38/base -> origin/gh/nivekt/38/base 2022-03-17T19:04:58.6997419Z * [new branch] gh/nivekt/38/head -> origin/gh/nivekt/38/head 2022-03-17T19:04:58.6998705Z * [new branch] gh/nivekt/38/orig -> origin/gh/nivekt/38/orig 2022-03-17T19:04:58.7000614Z * [new branch] gh/nivekt/41/base -> origin/gh/nivekt/41/base 2022-03-17T19:04:58.7001948Z * [new branch] gh/nivekt/41/head -> origin/gh/nivekt/41/head 2022-03-17T19:04:58.7003588Z * [new branch] gh/nivekt/41/orig -> origin/gh/nivekt/41/orig 2022-03-17T19:04:58.7005244Z * [new branch] gh/nivekt/42/base -> origin/gh/nivekt/42/base 2022-03-17T19:04:58.7006576Z * [new branch] gh/nivekt/42/head -> origin/gh/nivekt/42/head 2022-03-17T19:04:58.7007808Z * [new branch] gh/nivekt/42/orig -> origin/gh/nivekt/42/orig 2022-03-17T19:04:58.7009672Z * [new branch] gh/nivekt/49/base -> origin/gh/nivekt/49/base 2022-03-17T19:04:58.7011000Z * [new branch] gh/nivekt/49/head -> origin/gh/nivekt/49/head 2022-03-17T19:04:58.7012296Z * [new branch] gh/nivekt/49/orig -> origin/gh/nivekt/49/orig 2022-03-17T19:04:58.7014233Z * [new branch] gh/nivekt/52/base -> origin/gh/nivekt/52/base 2022-03-17T19:04:58.7015502Z * [new branch] gh/nivekt/52/head -> origin/gh/nivekt/52/head 2022-03-17T19:04:58.7017013Z * [new branch] gh/nivekt/52/orig -> origin/gh/nivekt/52/orig 2022-03-17T19:04:58.7018809Z * [new branch] gh/nivekt/8/base -> origin/gh/nivekt/8/base 2022-03-17T19:04:58.7020002Z * [new branch] gh/nivekt/8/head -> origin/gh/nivekt/8/head 2022-03-17T19:04:58.7021327Z * [new branch] gh/nivekt/8/orig -> origin/gh/nivekt/8/orig 2022-03-17T19:04:58.7023402Z * [new branch] gh/nkaretnikov/25/base -> origin/gh/nkaretnikov/25/base 2022-03-17T19:04:58.7024497Z * [new branch] gh/nkaretnikov/25/head -> origin/gh/nkaretnikov/25/head 2022-03-17T19:04:58.7025840Z * [new branch] gh/nkaretnikov/25/orig -> origin/gh/nkaretnikov/25/orig 2022-03-17T19:04:58.7027721Z * [new branch] gh/nkaretnikov/28/base -> origin/gh/nkaretnikov/28/base 2022-03-17T19:04:58.7028781Z * [new branch] gh/nkaretnikov/28/head -> origin/gh/nkaretnikov/28/head 2022-03-17T19:04:58.7030106Z * [new branch] gh/nkaretnikov/28/orig -> origin/gh/nkaretnikov/28/orig 2022-03-17T19:04:58.7031913Z * [new branch] gh/nkaretnikov/33/base -> origin/gh/nkaretnikov/33/base 2022-03-17T19:04:58.7033055Z * [new branch] gh/nkaretnikov/33/head -> origin/gh/nkaretnikov/33/head 2022-03-17T19:04:58.7034649Z * [new branch] gh/nkaretnikov/33/orig -> origin/gh/nkaretnikov/33/orig 2022-03-17T19:04:58.7036370Z * [new branch] gh/nkaretnikov/34/base -> origin/gh/nkaretnikov/34/base 2022-03-17T19:04:58.7037510Z * [new branch] gh/nkaretnikov/34/head -> origin/gh/nkaretnikov/34/head 2022-03-17T19:04:58.7038695Z * [new branch] gh/nkaretnikov/34/orig -> origin/gh/nkaretnikov/34/orig 2022-03-17T19:04:58.7040473Z * [new branch] gh/osalpekar/1/base -> origin/gh/osalpekar/1/base 2022-03-17T19:04:58.7042428Z * [new branch] gh/osalpekar/1/head -> origin/gh/osalpekar/1/head 2022-03-17T19:04:58.7044799Z * [new branch] gh/osalpekar/102/base -> origin/gh/osalpekar/102/base 2022-03-17T19:04:58.7046045Z * [new branch] gh/osalpekar/102/head -> origin/gh/osalpekar/102/head 2022-03-17T19:04:58.7047564Z * [new branch] gh/osalpekar/102/orig -> origin/gh/osalpekar/102/orig 2022-03-17T19:04:58.7049249Z * [new branch] gh/osalpekar/107/base -> origin/gh/osalpekar/107/base 2022-03-17T19:04:58.7050746Z * [new branch] gh/osalpekar/107/head -> origin/gh/osalpekar/107/head 2022-03-17T19:04:58.7052100Z * [new branch] gh/osalpekar/2/base -> origin/gh/osalpekar/2/base 2022-03-17T19:04:58.7053422Z * [new branch] gh/osalpekar/2/head -> origin/gh/osalpekar/2/head 2022-03-17T19:04:58.7054828Z * [new branch] gh/osalpekar/3/base -> origin/gh/osalpekar/3/base 2022-03-17T19:04:58.7056155Z * [new branch] gh/osalpekar/3/head -> origin/gh/osalpekar/3/head 2022-03-17T19:04:58.7057631Z * [new branch] gh/osalpekar/4/base -> origin/gh/osalpekar/4/base 2022-03-17T19:04:58.7058839Z * [new branch] gh/osalpekar/4/head -> origin/gh/osalpekar/4/head 2022-03-17T19:04:58.7060704Z * [new branch] gh/osalpekar/40/base -> origin/gh/osalpekar/40/base 2022-03-17T19:04:58.7061967Z * [new branch] gh/osalpekar/40/orig -> origin/gh/osalpekar/40/orig 2022-03-17T19:04:58.7063521Z * [new branch] gh/osalpekar/6/base -> origin/gh/osalpekar/6/base 2022-03-17T19:04:58.7064930Z * [new branch] gh/osalpekar/6/head -> origin/gh/osalpekar/6/head 2022-03-17T19:04:58.7066282Z * [new branch] gh/osalpekar/6/orig -> origin/gh/osalpekar/6/orig 2022-03-17T19:04:58.7068016Z * [new branch] gh/osalpekar/60/base -> origin/gh/osalpekar/60/base 2022-03-17T19:04:58.7069358Z * [new branch] gh/osalpekar/60/head -> origin/gh/osalpekar/60/head 2022-03-17T19:04:58.7070933Z * [new branch] gh/osalpekar/60/orig -> origin/gh/osalpekar/60/orig 2022-03-17T19:04:58.7072295Z * [new branch] gh/osalpekar/61/base -> origin/gh/osalpekar/61/base 2022-03-17T19:04:58.7073646Z * [new branch] gh/osalpekar/61/head -> origin/gh/osalpekar/61/head 2022-03-17T19:04:58.7074996Z * [new branch] gh/osalpekar/61/orig -> origin/gh/osalpekar/61/orig 2022-03-17T19:04:58.7076610Z * [new branch] gh/osalpekar/62/base -> origin/gh/osalpekar/62/base 2022-03-17T19:04:58.7077970Z * [new branch] gh/osalpekar/62/head -> origin/gh/osalpekar/62/head 2022-03-17T19:04:58.7079387Z * [new branch] gh/osalpekar/62/orig -> origin/gh/osalpekar/62/orig 2022-03-17T19:04:58.7080950Z * [new branch] gh/osalpekar/76/base -> origin/gh/osalpekar/76/base 2022-03-17T19:04:58.7082319Z * [new branch] gh/osalpekar/76/head -> origin/gh/osalpekar/76/head 2022-03-17T19:04:58.7083869Z * [new branch] gh/osalpekar/76/orig -> origin/gh/osalpekar/76/orig 2022-03-17T19:04:58.7085597Z * [new branch] gh/osalpekar/77/base -> origin/gh/osalpekar/77/base 2022-03-17T19:04:58.7086912Z * [new branch] gh/osalpekar/77/head -> origin/gh/osalpekar/77/head 2022-03-17T19:04:58.7088242Z * [new branch] gh/osalpekar/77/orig -> origin/gh/osalpekar/77/orig 2022-03-17T19:04:58.7089921Z * [new branch] gh/osalpekar/78/base -> origin/gh/osalpekar/78/base 2022-03-17T19:04:58.7091225Z * [new branch] gh/osalpekar/78/head -> origin/gh/osalpekar/78/head 2022-03-17T19:04:58.7092501Z * [new branch] gh/osalpekar/78/orig -> origin/gh/osalpekar/78/orig 2022-03-17T19:04:58.7094387Z * [new branch] gh/osalpekar/85/base -> origin/gh/osalpekar/85/base 2022-03-17T19:04:58.7095634Z * [new branch] gh/osalpekar/85/head -> origin/gh/osalpekar/85/head 2022-03-17T19:04:58.7096969Z * [new branch] gh/osalpekar/85/orig -> origin/gh/osalpekar/85/orig 2022-03-17T19:04:58.7098539Z * [new branch] gh/osalpekar/86/base -> origin/gh/osalpekar/86/base 2022-03-17T19:04:58.7099827Z * [new branch] gh/osalpekar/86/head -> origin/gh/osalpekar/86/head 2022-03-17T19:04:58.7101283Z * [new branch] gh/osalpekar/86/orig -> origin/gh/osalpekar/86/orig 2022-03-17T19:04:58.7102997Z * [new branch] gh/osalpekar/91/base -> origin/gh/osalpekar/91/base 2022-03-17T19:04:58.7104315Z * [new branch] gh/osalpekar/91/head -> origin/gh/osalpekar/91/head 2022-03-17T19:04:58.7105683Z * [new branch] gh/osalpekar/91/orig -> origin/gh/osalpekar/91/orig 2022-03-17T19:04:58.7107680Z * [new branch] gh/pavithranrao/1/base -> origin/gh/pavithranrao/1/base 2022-03-17T19:04:58.7108951Z * [new branch] gh/pavithranrao/1/head -> origin/gh/pavithranrao/1/head 2022-03-17T19:04:58.7110640Z * [new branch] gh/pavithranrao/10/base -> origin/gh/pavithranrao/10/base 2022-03-17T19:04:58.7111760Z * [new branch] gh/pavithranrao/10/head -> origin/gh/pavithranrao/10/head 2022-03-17T19:04:58.7113234Z * [new branch] gh/pavithranrao/11/base -> origin/gh/pavithranrao/11/base 2022-03-17T19:04:58.7114484Z * [new branch] gh/pavithranrao/11/head -> origin/gh/pavithranrao/11/head 2022-03-17T19:04:58.7116273Z * [new branch] gh/pavithranrao/17/base -> origin/gh/pavithranrao/17/base 2022-03-17T19:04:58.7117483Z * [new branch] gh/pavithranrao/17/head -> origin/gh/pavithranrao/17/head 2022-03-17T19:04:58.7119745Z * [new branch] gh/pavithranrao/17/orig -> origin/gh/pavithranrao/17/orig 2022-03-17T19:04:58.7120518Z * [new branch] gh/pavithranrao/19/base -> origin/gh/pavithranrao/19/base 2022-03-17T19:04:58.7122329Z * [new branch] gh/pavithranrao/19/head -> origin/gh/pavithranrao/19/head 2022-03-17T19:04:58.7124991Z * [new branch] gh/pavithranrao/19/orig -> origin/gh/pavithranrao/19/orig 2022-03-17T19:04:58.7126062Z * [new branch] gh/pavithranrao/2/base -> origin/gh/pavithranrao/2/base 2022-03-17T19:04:58.7127452Z * [new branch] gh/pavithranrao/2/head -> origin/gh/pavithranrao/2/head 2022-03-17T19:04:58.7129796Z * [new branch] gh/pavithranrao/20/base -> origin/gh/pavithranrao/20/base 2022-03-17T19:04:58.7130479Z * [new branch] gh/pavithranrao/20/head -> origin/gh/pavithranrao/20/head 2022-03-17T19:04:58.7131567Z * [new branch] gh/pavithranrao/20/orig -> origin/gh/pavithranrao/20/orig 2022-03-17T19:04:58.7133403Z * [new branch] gh/pavithranrao/21/base -> origin/gh/pavithranrao/21/base 2022-03-17T19:04:58.7134614Z * [new branch] gh/pavithranrao/21/head -> origin/gh/pavithranrao/21/head 2022-03-17T19:04:58.7136247Z * [new branch] gh/pavithranrao/21/orig -> origin/gh/pavithranrao/21/orig 2022-03-17T19:04:58.7137757Z * [new branch] gh/pavithranrao/25/base -> origin/gh/pavithranrao/25/base 2022-03-17T19:04:58.7138974Z * [new branch] gh/pavithranrao/25/head -> origin/gh/pavithranrao/25/head 2022-03-17T19:04:58.7140539Z * [new branch] gh/pavithranrao/25/orig -> origin/gh/pavithranrao/25/orig 2022-03-17T19:04:58.7142096Z * [new branch] gh/pavithranrao/26/base -> origin/gh/pavithranrao/26/base 2022-03-17T19:04:58.7143296Z * [new branch] gh/pavithranrao/26/head -> origin/gh/pavithranrao/26/head 2022-03-17T19:04:58.7145221Z * [new branch] gh/pavithranrao/28/base -> origin/gh/pavithranrao/28/base 2022-03-17T19:04:58.7146449Z * [new branch] gh/pavithranrao/28/head -> origin/gh/pavithranrao/28/head 2022-03-17T19:04:58.7148065Z * [new branch] gh/pavithranrao/29/base -> origin/gh/pavithranrao/29/base 2022-03-17T19:04:58.7149256Z * [new branch] gh/pavithranrao/29/head -> origin/gh/pavithranrao/29/head 2022-03-17T19:04:58.7150911Z * [new branch] gh/pavithranrao/3/base -> origin/gh/pavithranrao/3/base 2022-03-17T19:04:58.7152032Z * [new branch] gh/pavithranrao/3/head -> origin/gh/pavithranrao/3/head 2022-03-17T19:04:58.7153661Z * [new branch] gh/pavithranrao/30/base -> origin/gh/pavithranrao/30/base 2022-03-17T19:04:58.7154872Z * [new branch] gh/pavithranrao/30/head -> origin/gh/pavithranrao/30/head 2022-03-17T19:04:58.7156496Z * [new branch] gh/pavithranrao/31/base -> origin/gh/pavithranrao/31/base 2022-03-17T19:04:58.7157577Z * [new branch] gh/pavithranrao/31/head -> origin/gh/pavithranrao/31/head 2022-03-17T19:04:58.7159311Z * [new branch] gh/pavithranrao/32/base -> origin/gh/pavithranrao/32/base 2022-03-17T19:04:58.7160320Z * [new branch] gh/pavithranrao/32/head -> origin/gh/pavithranrao/32/head 2022-03-17T19:04:58.7162241Z * [new branch] gh/pavithranrao/33/base -> origin/gh/pavithranrao/33/base 2022-03-17T19:04:58.7163442Z * [new branch] gh/pavithranrao/33/head -> origin/gh/pavithranrao/33/head 2022-03-17T19:04:58.7165093Z * [new branch] gh/pavithranrao/33/orig -> origin/gh/pavithranrao/33/orig 2022-03-17T19:04:58.7166841Z * [new branch] gh/pavithranrao/34/base -> origin/gh/pavithranrao/34/base 2022-03-17T19:04:58.7167959Z * [new branch] gh/pavithranrao/34/head -> origin/gh/pavithranrao/34/head 2022-03-17T19:04:58.7169462Z * [new branch] gh/pavithranrao/34/orig -> origin/gh/pavithranrao/34/orig 2022-03-17T19:04:58.7171153Z * [new branch] gh/pavithranrao/36/base -> origin/gh/pavithranrao/36/base 2022-03-17T19:04:58.7172310Z * [new branch] gh/pavithranrao/36/head -> origin/gh/pavithranrao/36/head 2022-03-17T19:04:58.7174062Z * [new branch] gh/pavithranrao/36/orig -> origin/gh/pavithranrao/36/orig 2022-03-17T19:04:58.7175753Z * [new branch] gh/pavithranrao/37/base -> origin/gh/pavithranrao/37/base 2022-03-17T19:04:58.7176932Z * [new branch] gh/pavithranrao/37/head -> origin/gh/pavithranrao/37/head 2022-03-17T19:04:58.7178434Z * [new branch] gh/pavithranrao/37/orig -> origin/gh/pavithranrao/37/orig 2022-03-17T19:04:58.7179955Z * [new branch] gh/pavithranrao/39/base -> origin/gh/pavithranrao/39/base 2022-03-17T19:04:58.7181166Z * [new branch] gh/pavithranrao/39/head -> origin/gh/pavithranrao/39/head 2022-03-17T19:04:58.7182696Z * [new branch] gh/pavithranrao/39/orig -> origin/gh/pavithranrao/39/orig 2022-03-17T19:04:58.7183983Z * [new branch] gh/pavithranrao/4/base -> origin/gh/pavithranrao/4/base 2022-03-17T19:04:58.7185489Z * [new branch] gh/pavithranrao/4/head -> origin/gh/pavithranrao/4/head 2022-03-17T19:04:58.7187291Z * [new branch] gh/pavithranrao/40/base -> origin/gh/pavithranrao/40/base 2022-03-17T19:04:58.7188382Z * [new branch] gh/pavithranrao/40/head -> origin/gh/pavithranrao/40/head 2022-03-17T19:04:58.7190032Z * [new branch] gh/pavithranrao/40/orig -> origin/gh/pavithranrao/40/orig 2022-03-17T19:04:58.7191802Z * [new branch] gh/pavithranrao/41/base -> origin/gh/pavithranrao/41/base 2022-03-17T19:04:58.7192851Z * [new branch] gh/pavithranrao/41/head -> origin/gh/pavithranrao/41/head 2022-03-17T19:04:58.7194585Z * [new branch] gh/pavithranrao/42/base -> origin/gh/pavithranrao/42/base 2022-03-17T19:04:58.7195633Z * [new branch] gh/pavithranrao/42/head -> origin/gh/pavithranrao/42/head 2022-03-17T19:04:58.7197320Z * [new branch] gh/pavithranrao/42/orig -> origin/gh/pavithranrao/42/orig 2022-03-17T19:04:58.7199007Z * [new branch] gh/pavithranrao/45/base -> origin/gh/pavithranrao/45/base 2022-03-17T19:04:58.7199982Z * [new branch] gh/pavithranrao/45/head -> origin/gh/pavithranrao/45/head 2022-03-17T19:04:58.7201340Z * [new branch] gh/pavithranrao/45/orig -> origin/gh/pavithranrao/45/orig 2022-03-17T19:04:58.7204038Z * [new branch] gh/pavithranrao/46/base -> origin/gh/pavithranrao/46/base 2022-03-17T19:04:58.7205324Z * [new branch] gh/pavithranrao/46/head -> origin/gh/pavithranrao/46/head 2022-03-17T19:04:58.7206811Z * [new branch] gh/pavithranrao/47/base -> origin/gh/pavithranrao/47/base 2022-03-17T19:04:58.7208248Z * [new branch] gh/pavithranrao/47/head -> origin/gh/pavithranrao/47/head 2022-03-17T19:04:58.7209668Z * [new branch] gh/pavithranrao/47/orig -> origin/gh/pavithranrao/47/orig 2022-03-17T19:04:58.7211296Z * [new branch] gh/pavithranrao/48/base -> origin/gh/pavithranrao/48/base 2022-03-17T19:04:58.7212683Z * [new branch] gh/pavithranrao/48/head -> origin/gh/pavithranrao/48/head 2022-03-17T19:04:58.7214016Z * [new branch] gh/pavithranrao/48/orig -> origin/gh/pavithranrao/48/orig 2022-03-17T19:04:58.7215412Z * [new branch] gh/pavithranrao/5/base -> origin/gh/pavithranrao/5/base 2022-03-17T19:04:58.7216800Z * [new branch] gh/pavithranrao/5/head -> origin/gh/pavithranrao/5/head 2022-03-17T19:04:58.7218497Z * [new branch] gh/pavithranrao/50/base -> origin/gh/pavithranrao/50/base 2022-03-17T19:04:58.7219786Z * [new branch] gh/pavithranrao/50/head -> origin/gh/pavithranrao/50/head 2022-03-17T19:04:58.7221126Z * [new branch] gh/pavithranrao/50/orig -> origin/gh/pavithranrao/50/orig 2022-03-17T19:04:58.7222851Z * [new branch] gh/pavithranrao/51/base -> origin/gh/pavithranrao/51/base 2022-03-17T19:04:58.7224204Z * [new branch] gh/pavithranrao/51/head -> origin/gh/pavithranrao/51/head 2022-03-17T19:04:58.7225834Z * [new branch] gh/pavithranrao/51/orig -> origin/gh/pavithranrao/51/orig 2022-03-17T19:04:58.7227369Z * [new branch] gh/pavithranrao/6/base -> origin/gh/pavithranrao/6/base 2022-03-17T19:04:58.7228511Z * [new branch] gh/pavithranrao/6/head -> origin/gh/pavithranrao/6/head 2022-03-17T19:04:58.7230013Z * [new branch] gh/pavithranrao/7/base -> origin/gh/pavithranrao/7/base 2022-03-17T19:04:58.7231315Z * [new branch] gh/pavithranrao/7/head -> origin/gh/pavithranrao/7/head 2022-03-17T19:04:58.7232800Z * [new branch] gh/pavithranrao/8/base -> origin/gh/pavithranrao/8/base 2022-03-17T19:04:58.7234082Z * [new branch] gh/pavithranrao/8/head -> origin/gh/pavithranrao/8/head 2022-03-17T19:04:58.7235972Z * [new branch] gh/pbelevich/142/base -> origin/gh/pbelevich/142/base 2022-03-17T19:04:58.7237400Z * [new branch] gh/pbelevich/142/head -> origin/gh/pbelevich/142/head 2022-03-17T19:04:58.7238757Z * [new branch] gh/pbelevich/142/orig -> origin/gh/pbelevich/142/orig 2022-03-17T19:04:58.7240243Z * [new branch] gh/pbelevich/143/base -> origin/gh/pbelevich/143/base 2022-03-17T19:04:58.7241628Z * [new branch] gh/pbelevich/143/head -> origin/gh/pbelevich/143/head 2022-03-17T19:04:58.7243311Z * [new branch] gh/pbelevich/143/orig -> origin/gh/pbelevich/143/orig 2022-03-17T19:04:58.7244744Z * [new branch] gh/pbelevich/144/base -> origin/gh/pbelevich/144/base 2022-03-17T19:04:58.7246115Z * [new branch] gh/pbelevich/144/head -> origin/gh/pbelevich/144/head 2022-03-17T19:04:58.7247981Z * [new branch] gh/pbelevich/144/orig -> origin/gh/pbelevich/144/orig 2022-03-17T19:04:58.7249606Z * [new branch] gh/pbelevich/146/base -> origin/gh/pbelevich/146/base 2022-03-17T19:04:58.7251114Z * [new branch] gh/pbelevich/146/head -> origin/gh/pbelevich/146/head 2022-03-17T19:04:58.7252363Z * [new branch] gh/pbelevich/146/orig -> origin/gh/pbelevich/146/orig 2022-03-17T19:04:58.7254174Z * [new branch] gh/pbelevich/153/base -> origin/gh/pbelevich/153/base 2022-03-17T19:04:58.7255485Z * [new branch] gh/pbelevich/153/head -> origin/gh/pbelevich/153/head 2022-03-17T19:04:58.7256773Z * [new branch] gh/pbelevich/153/orig -> origin/gh/pbelevich/153/orig 2022-03-17T19:04:58.7258641Z * [new branch] gh/pbelevich/155/base -> origin/gh/pbelevich/155/base 2022-03-17T19:04:58.7259887Z * [new branch] gh/pbelevich/155/head -> origin/gh/pbelevich/155/head 2022-03-17T19:04:58.7261218Z * [new branch] gh/pbelevich/155/orig -> origin/gh/pbelevich/155/orig 2022-03-17T19:04:58.7262971Z * [new branch] gh/pbelevich/157/base -> origin/gh/pbelevich/157/base 2022-03-17T19:04:58.7264221Z * [new branch] gh/pbelevich/157/head -> origin/gh/pbelevich/157/head 2022-03-17T19:04:58.7265537Z * [new branch] gh/pbelevich/157/orig -> origin/gh/pbelevich/157/orig 2022-03-17T19:04:58.7267243Z * [new branch] gh/pbelevich/82/head -> origin/gh/pbelevich/82/head 2022-03-17T19:04:58.7268498Z * [new branch] gh/pbelevich/82/orig -> origin/gh/pbelevich/82/orig 2022-03-17T19:04:58.7270502Z * [new branch] gh/pearu/20/base -> origin/gh/pearu/20/base 2022-03-17T19:04:58.7272036Z * [new branch] gh/pearu/20/head -> origin/gh/pearu/20/head 2022-03-17T19:04:58.7273329Z * [new branch] gh/pearu/20/orig -> origin/gh/pearu/20/orig 2022-03-17T19:04:58.7275076Z * [new branch] gh/pearu/22/base -> origin/gh/pearu/22/base 2022-03-17T19:04:58.7276271Z * [new branch] gh/pearu/22/head -> origin/gh/pearu/22/head 2022-03-17T19:04:58.7277805Z * [new branch] gh/pearu/22/orig -> origin/gh/pearu/22/orig 2022-03-17T19:04:58.7279123Z * [new branch] gh/pearu/24/base -> origin/gh/pearu/24/base 2022-03-17T19:04:58.7280453Z * [new branch] gh/pearu/24/head -> origin/gh/pearu/24/head 2022-03-17T19:04:58.7281798Z * [new branch] gh/pearu/24/orig -> origin/gh/pearu/24/orig 2022-03-17T19:04:58.7283848Z * [new branch] gh/pearu/25/base -> origin/gh/pearu/25/base 2022-03-17T19:04:58.7285030Z * [new branch] gh/pearu/25/head -> origin/gh/pearu/25/head 2022-03-17T19:04:58.7286476Z * [new branch] gh/pearu/25/orig -> origin/gh/pearu/25/orig 2022-03-17T19:04:58.7288131Z * [new branch] gh/pearu/32/base -> origin/gh/pearu/32/base 2022-03-17T19:04:58.7289397Z * [new branch] gh/pearu/32/head -> origin/gh/pearu/32/head 2022-03-17T19:04:58.7290890Z * [new branch] gh/pearu/32/orig -> origin/gh/pearu/32/orig 2022-03-17T19:04:58.7292553Z * [new branch] gh/pearu/35/base -> origin/gh/pearu/35/base 2022-03-17T19:04:58.7293875Z * [new branch] gh/pearu/35/head -> origin/gh/pearu/35/head 2022-03-17T19:04:58.7295158Z * [new branch] gh/pearu/35/orig -> origin/gh/pearu/35/orig 2022-03-17T19:04:58.7296944Z * [new branch] gh/pearu/41/base -> origin/gh/pearu/41/base 2022-03-17T19:04:58.7298172Z * [new branch] gh/pearu/41/head -> origin/gh/pearu/41/head 2022-03-17T19:04:58.7299537Z * [new branch] gh/pearu/41/orig -> origin/gh/pearu/41/orig 2022-03-17T19:04:58.7301612Z * [new branch] gh/penguinwu/10/base -> origin/gh/penguinwu/10/base 2022-03-17T19:04:58.7302940Z * [new branch] gh/penguinwu/10/head -> origin/gh/penguinwu/10/head 2022-03-17T19:04:58.7304341Z * [new branch] gh/penguinwu/10/orig -> origin/gh/penguinwu/10/orig 2022-03-17T19:04:58.7305934Z * [new branch] gh/penguinwu/11/base -> origin/gh/penguinwu/11/base 2022-03-17T19:04:58.7307203Z * [new branch] gh/penguinwu/11/head -> origin/gh/penguinwu/11/head 2022-03-17T19:04:58.7308575Z * [new branch] gh/penguinwu/11/orig -> origin/gh/penguinwu/11/orig 2022-03-17T19:04:58.7310322Z * [new branch] gh/penguinwu/7/base -> origin/gh/penguinwu/7/base 2022-03-17T19:04:58.7311612Z * [new branch] gh/penguinwu/7/head -> origin/gh/penguinwu/7/head 2022-03-17T19:04:58.7312932Z * [new branch] gh/penguinwu/7/orig -> origin/gh/penguinwu/7/orig 2022-03-17T19:04:58.7314923Z * [new branch] gh/peterbell10/100/base -> origin/gh/peterbell10/100/base 2022-03-17T19:04:58.7316257Z * [new branch] gh/peterbell10/100/orig -> origin/gh/peterbell10/100/orig 2022-03-17T19:04:58.7318081Z * [new branch] gh/peterbell10/107/orig -> origin/gh/peterbell10/107/orig 2022-03-17T19:04:58.7319738Z * [new branch] gh/peterbell10/109/base -> origin/gh/peterbell10/109/base 2022-03-17T19:04:58.7321010Z * [new branch] gh/peterbell10/109/head -> origin/gh/peterbell10/109/head 2022-03-17T19:04:58.7322431Z * [new branch] gh/peterbell10/109/orig -> origin/gh/peterbell10/109/orig 2022-03-17T19:04:58.7324355Z * [new branch] gh/peterbell10/112/base -> origin/gh/peterbell10/112/base 2022-03-17T19:04:58.7325483Z * [new branch] gh/peterbell10/112/head -> origin/gh/peterbell10/112/head 2022-03-17T19:04:58.7326532Z * [new branch] gh/peterbell10/112/orig -> origin/gh/peterbell10/112/orig 2022-03-17T19:04:58.7328878Z * [new branch] gh/peterbell10/157/base -> origin/gh/peterbell10/157/base 2022-03-17T19:04:58.7329857Z * [new branch] gh/peterbell10/157/head -> origin/gh/peterbell10/157/head 2022-03-17T19:04:58.7331674Z * [new branch] gh/peterbell10/157/orig -> origin/gh/peterbell10/157/orig 2022-03-17T19:04:58.7332802Z * [new branch] gh/peterbell10/158/base -> origin/gh/peterbell10/158/base 2022-03-17T19:04:58.7334414Z * [new branch] gh/peterbell10/158/head -> origin/gh/peterbell10/158/head 2022-03-17T19:04:58.7335404Z * [new branch] gh/peterbell10/158/orig -> origin/gh/peterbell10/158/orig 2022-03-17T19:04:58.7337433Z * [new branch] gh/peterbell10/161/base -> origin/gh/peterbell10/161/base 2022-03-17T19:04:58.7338532Z * [new branch] gh/peterbell10/161/head -> origin/gh/peterbell10/161/head 2022-03-17T19:04:58.7340147Z * [new branch] gh/peterbell10/161/orig -> origin/gh/peterbell10/161/orig 2022-03-17T19:04:58.7341746Z * [new branch] gh/peterbell10/174/base -> origin/gh/peterbell10/174/base 2022-03-17T19:04:58.7342987Z * [new branch] gh/peterbell10/174/head -> origin/gh/peterbell10/174/head 2022-03-17T19:04:58.7344440Z * [new branch] gh/peterbell10/174/orig -> origin/gh/peterbell10/174/orig 2022-03-17T19:04:58.7346472Z * [new branch] gh/peterbell10/175/base -> origin/gh/peterbell10/175/base 2022-03-17T19:04:58.7347625Z * [new branch] gh/peterbell10/175/head -> origin/gh/peterbell10/175/head 2022-03-17T19:04:58.7349234Z * [new branch] gh/peterbell10/175/orig -> origin/gh/peterbell10/175/orig 2022-03-17T19:04:58.7350953Z * [new branch] gh/peterbell10/18/base -> origin/gh/peterbell10/18/base 2022-03-17T19:04:58.7352106Z * [new branch] gh/peterbell10/18/head -> origin/gh/peterbell10/18/head 2022-03-17T19:04:58.7353604Z * [new branch] gh/peterbell10/18/orig -> origin/gh/peterbell10/18/orig 2022-03-17T19:04:58.7355322Z * [new branch] gh/peterbell10/181/base -> origin/gh/peterbell10/181/base 2022-03-17T19:04:58.7356432Z * [new branch] gh/peterbell10/181/head -> origin/gh/peterbell10/181/head 2022-03-17T19:04:58.7357943Z * [new branch] gh/peterbell10/181/orig -> origin/gh/peterbell10/181/orig 2022-03-17T19:04:58.7359746Z * [new branch] gh/peterbell10/186/base -> origin/gh/peterbell10/186/base 2022-03-17T19:04:58.7360924Z * [new branch] gh/peterbell10/186/head -> origin/gh/peterbell10/186/head 2022-03-17T19:04:58.7362597Z * [new branch] gh/peterbell10/186/orig -> origin/gh/peterbell10/186/orig 2022-03-17T19:04:58.7364769Z * [new branch] gh/peterbell10/187/base -> origin/gh/peterbell10/187/base 2022-03-17T19:04:58.7365921Z * [new branch] gh/peterbell10/187/head -> origin/gh/peterbell10/187/head 2022-03-17T19:04:58.7367555Z * [new branch] gh/peterbell10/187/orig -> origin/gh/peterbell10/187/orig 2022-03-17T19:04:58.7369348Z * [new branch] gh/peterbell10/188/base -> origin/gh/peterbell10/188/base 2022-03-17T19:04:58.7370590Z * [new branch] gh/peterbell10/188/head -> origin/gh/peterbell10/188/head 2022-03-17T19:04:58.7371807Z * [new branch] gh/peterbell10/188/orig -> origin/gh/peterbell10/188/orig 2022-03-17T19:04:58.7373774Z * [new branch] gh/peterbell10/189/base -> origin/gh/peterbell10/189/base 2022-03-17T19:04:58.7375036Z * [new branch] gh/peterbell10/189/head -> origin/gh/peterbell10/189/head 2022-03-17T19:04:58.7379591Z * [new branch] gh/peterbell10/189/orig -> origin/gh/peterbell10/189/orig 2022-03-17T19:04:58.7380183Z * [new branch] gh/peterbell10/202/base -> origin/gh/peterbell10/202/base 2022-03-17T19:04:58.7380764Z * [new branch] gh/peterbell10/202/head -> origin/gh/peterbell10/202/head 2022-03-17T19:04:58.7381337Z * [new branch] gh/peterbell10/202/orig -> origin/gh/peterbell10/202/orig 2022-03-17T19:04:58.7382573Z * [new branch] gh/peterbell10/206/base -> origin/gh/peterbell10/206/base 2022-03-17T19:04:58.7384246Z * [new branch] gh/peterbell10/206/head -> origin/gh/peterbell10/206/head 2022-03-17T19:04:58.7385280Z * [new branch] gh/peterbell10/206/orig -> origin/gh/peterbell10/206/orig 2022-03-17T19:04:58.7387183Z * [new branch] gh/peterbell10/207/base -> origin/gh/peterbell10/207/base 2022-03-17T19:04:58.7388283Z * [new branch] gh/peterbell10/207/head -> origin/gh/peterbell10/207/head 2022-03-17T19:04:58.7390246Z * [new branch] gh/peterbell10/207/orig -> origin/gh/peterbell10/207/orig 2022-03-17T19:04:58.7391312Z * [new branch] gh/peterbell10/208/base -> origin/gh/peterbell10/208/base 2022-03-17T19:04:58.7392555Z * [new branch] gh/peterbell10/208/head -> origin/gh/peterbell10/208/head 2022-03-17T19:04:58.7394133Z * [new branch] gh/peterbell10/208/orig -> origin/gh/peterbell10/208/orig 2022-03-17T19:04:58.7395926Z * [new branch] gh/peterbell10/209/base -> origin/gh/peterbell10/209/base 2022-03-17T19:04:58.7397060Z * [new branch] gh/peterbell10/209/head -> origin/gh/peterbell10/209/head 2022-03-17T19:04:58.7398306Z * [new branch] gh/peterbell10/209/orig -> origin/gh/peterbell10/209/orig 2022-03-17T19:04:58.7400283Z * [new branch] gh/peterbell10/213/base -> origin/gh/peterbell10/213/base 2022-03-17T19:04:58.7401347Z * [new branch] gh/peterbell10/213/head -> origin/gh/peterbell10/213/head 2022-03-17T19:04:58.7402951Z * [new branch] gh/peterbell10/213/orig -> origin/gh/peterbell10/213/orig 2022-03-17T19:04:58.7404727Z * [new branch] gh/peterbell10/216/base -> origin/gh/peterbell10/216/base 2022-03-17T19:04:58.7405806Z * [new branch] gh/peterbell10/216/head -> origin/gh/peterbell10/216/head 2022-03-17T19:04:58.7407277Z * [new branch] gh/peterbell10/216/orig -> origin/gh/peterbell10/216/orig 2022-03-17T19:04:58.7409032Z * [new branch] gh/peterbell10/253/base -> origin/gh/peterbell10/253/base 2022-03-17T19:04:58.7410551Z * [new branch] gh/peterbell10/253/head -> origin/gh/peterbell10/253/head 2022-03-17T19:04:58.7411709Z * [new branch] gh/peterbell10/253/orig -> origin/gh/peterbell10/253/orig 2022-03-17T19:04:58.7413434Z * [new branch] gh/peterbell10/254/base -> origin/gh/peterbell10/254/base 2022-03-17T19:04:58.7414668Z * [new branch] gh/peterbell10/254/head -> origin/gh/peterbell10/254/head 2022-03-17T19:04:58.7416118Z * [new branch] gh/peterbell10/254/orig -> origin/gh/peterbell10/254/orig 2022-03-17T19:04:58.7417688Z * [new branch] gh/peterbell10/255/base -> origin/gh/peterbell10/255/base 2022-03-17T19:04:58.7418819Z * [new branch] gh/peterbell10/255/head -> origin/gh/peterbell10/255/head 2022-03-17T19:04:58.7420321Z * [new branch] gh/peterbell10/255/orig -> origin/gh/peterbell10/255/orig 2022-03-17T19:04:58.7422223Z * [new branch] gh/peterbell10/256/base -> origin/gh/peterbell10/256/base 2022-03-17T19:04:58.7423659Z * [new branch] gh/peterbell10/256/head -> origin/gh/peterbell10/256/head 2022-03-17T19:04:58.7424843Z * [new branch] gh/peterbell10/256/orig -> origin/gh/peterbell10/256/orig 2022-03-17T19:04:58.7426534Z * [new branch] gh/peterbell10/257/base -> origin/gh/peterbell10/257/base 2022-03-17T19:04:58.7427664Z * [new branch] gh/peterbell10/257/head -> origin/gh/peterbell10/257/head 2022-03-17T19:04:58.7429094Z * [new branch] gh/peterbell10/257/orig -> origin/gh/peterbell10/257/orig 2022-03-17T19:04:58.7430829Z * [new branch] gh/peterbell10/267/base -> origin/gh/peterbell10/267/base 2022-03-17T19:04:58.7432063Z * [new branch] gh/peterbell10/267/head -> origin/gh/peterbell10/267/head 2022-03-17T19:04:58.7433514Z * [new branch] gh/peterbell10/267/orig -> origin/gh/peterbell10/267/orig 2022-03-17T19:04:58.7435311Z * [new branch] gh/peterbell10/268/base -> origin/gh/peterbell10/268/base 2022-03-17T19:04:58.7436407Z * [new branch] gh/peterbell10/268/head -> origin/gh/peterbell10/268/head 2022-03-17T19:04:58.7437849Z * [new branch] gh/peterbell10/268/orig -> origin/gh/peterbell10/268/orig 2022-03-17T19:04:58.7439524Z * [new branch] gh/peterbell10/273/base -> origin/gh/peterbell10/273/base 2022-03-17T19:04:58.7441018Z * [new branch] gh/peterbell10/273/head -> origin/gh/peterbell10/273/head 2022-03-17T19:04:58.7442379Z * [new branch] gh/peterbell10/273/orig -> origin/gh/peterbell10/273/orig 2022-03-17T19:04:58.7444281Z * [new branch] gh/peterbell10/277/base -> origin/gh/peterbell10/277/base 2022-03-17T19:04:58.7445494Z * [new branch] gh/peterbell10/277/head -> origin/gh/peterbell10/277/head 2022-03-17T19:04:58.7447051Z * [new branch] gh/peterbell10/277/orig -> origin/gh/peterbell10/277/orig 2022-03-17T19:04:58.7448818Z * [new branch] gh/peterbell10/278/base -> origin/gh/peterbell10/278/base 2022-03-17T19:04:58.7449725Z * [new branch] gh/peterbell10/278/head -> origin/gh/peterbell10/278/head 2022-03-17T19:04:58.7451744Z * [new branch] gh/peterbell10/278/orig -> origin/gh/peterbell10/278/orig 2022-03-17T19:04:58.7453293Z * [new branch] gh/peterbell10/279/base -> origin/gh/peterbell10/279/base 2022-03-17T19:04:58.7454623Z * [new branch] gh/peterbell10/279/head -> origin/gh/peterbell10/279/head 2022-03-17T19:04:58.7455932Z * [new branch] gh/peterbell10/279/orig -> origin/gh/peterbell10/279/orig 2022-03-17T19:04:58.7457643Z * [new branch] gh/peterbell10/280/base -> origin/gh/peterbell10/280/base 2022-03-17T19:04:58.7458912Z * [new branch] gh/peterbell10/280/head -> origin/gh/peterbell10/280/head 2022-03-17T19:04:58.7460284Z * [new branch] gh/peterbell10/280/orig -> origin/gh/peterbell10/280/orig 2022-03-17T19:04:58.7461990Z * [new branch] gh/peterbell10/281/base -> origin/gh/peterbell10/281/base 2022-03-17T19:04:58.7463244Z * [new branch] gh/peterbell10/281/head -> origin/gh/peterbell10/281/head 2022-03-17T19:04:58.7464553Z * [new branch] gh/peterbell10/281/orig -> origin/gh/peterbell10/281/orig 2022-03-17T19:04:58.7466291Z * [new branch] gh/peterbell10/282/base -> origin/gh/peterbell10/282/base 2022-03-17T19:04:58.7467510Z * [new branch] gh/peterbell10/282/head -> origin/gh/peterbell10/282/head 2022-03-17T19:04:58.7468913Z * [new branch] gh/peterbell10/282/orig -> origin/gh/peterbell10/282/orig 2022-03-17T19:04:58.7470781Z * [new branch] gh/peterbell10/283/base -> origin/gh/peterbell10/283/base 2022-03-17T19:04:58.7471981Z * [new branch] gh/peterbell10/283/head -> origin/gh/peterbell10/283/head 2022-03-17T19:04:58.7473223Z * [new branch] gh/peterbell10/283/orig -> origin/gh/peterbell10/283/orig 2022-03-17T19:04:58.7475085Z * [new branch] gh/peterbell10/284/base -> origin/gh/peterbell10/284/base 2022-03-17T19:04:58.7476256Z * [new branch] gh/peterbell10/284/head -> origin/gh/peterbell10/284/head 2022-03-17T19:04:58.7477462Z * [new branch] gh/peterbell10/284/orig -> origin/gh/peterbell10/284/orig 2022-03-17T19:04:58.7479335Z * [new branch] gh/peterbell10/285/base -> origin/gh/peterbell10/285/base 2022-03-17T19:04:58.7480499Z * [new branch] gh/peterbell10/285/head -> origin/gh/peterbell10/285/head 2022-03-17T19:04:58.7481700Z * [new branch] gh/peterbell10/285/orig -> origin/gh/peterbell10/285/orig 2022-03-17T19:04:58.7483729Z * [new branch] gh/peterbell10/39/base -> origin/gh/peterbell10/39/base 2022-03-17T19:04:58.7485163Z * [new branch] gh/peterbell10/39/head -> origin/gh/peterbell10/39/head 2022-03-17T19:04:58.7486371Z * [new branch] gh/peterbell10/39/orig -> origin/gh/peterbell10/39/orig 2022-03-17T19:04:58.7488045Z * [new branch] gh/peterbell10/53/base -> origin/gh/peterbell10/53/base 2022-03-17T19:04:58.7489245Z * [new branch] gh/peterbell10/53/head -> origin/gh/peterbell10/53/head 2022-03-17T19:04:58.7490582Z * [new branch] gh/peterbell10/53/orig -> origin/gh/peterbell10/53/orig 2022-03-17T19:04:58.7492322Z * [new branch] gh/peterbell10/58/base -> origin/gh/peterbell10/58/base 2022-03-17T19:04:58.7493699Z * [new branch] gh/peterbell10/58/head -> origin/gh/peterbell10/58/head 2022-03-17T19:04:58.7495002Z * [new branch] gh/peterbell10/58/orig -> origin/gh/peterbell10/58/orig 2022-03-17T19:04:58.7496728Z * [new branch] gh/peterbell10/59/base -> origin/gh/peterbell10/59/base 2022-03-17T19:04:58.7498028Z * [new branch] gh/peterbell10/59/head -> origin/gh/peterbell10/59/head 2022-03-17T19:04:58.7499392Z * [new branch] gh/peterbell10/59/orig -> origin/gh/peterbell10/59/orig 2022-03-17T19:04:58.7501114Z * [new branch] gh/peterbell10/60/base -> origin/gh/peterbell10/60/base 2022-03-17T19:04:58.7502402Z * [new branch] gh/peterbell10/60/head -> origin/gh/peterbell10/60/head 2022-03-17T19:04:58.7503634Z * [new branch] gh/peterbell10/60/orig -> origin/gh/peterbell10/60/orig 2022-03-17T19:04:58.7505533Z * [new branch] gh/peterbell10/61/base -> origin/gh/peterbell10/61/base 2022-03-17T19:04:58.7506791Z * [new branch] gh/peterbell10/61/head -> origin/gh/peterbell10/61/head 2022-03-17T19:04:58.7508072Z * [new branch] gh/peterbell10/61/orig -> origin/gh/peterbell10/61/orig 2022-03-17T19:04:58.7509789Z * [new branch] gh/peterbell10/66/base -> origin/gh/peterbell10/66/base 2022-03-17T19:04:58.7511036Z * [new branch] gh/peterbell10/66/head -> origin/gh/peterbell10/66/head 2022-03-17T19:04:58.7512354Z * [new branch] gh/peterbell10/66/orig -> origin/gh/peterbell10/66/orig 2022-03-17T19:04:58.7514333Z * [new branch] gh/pietern/10/base -> origin/gh/pietern/10/base 2022-03-17T19:04:58.7515663Z * [new branch] gh/pietern/10/head -> origin/gh/pietern/10/head 2022-03-17T19:04:58.7517052Z * [new branch] gh/pietern/10/orig -> origin/gh/pietern/10/orig 2022-03-17T19:04:58.7518769Z * [new branch] gh/pietern/23/base -> origin/gh/pietern/23/base 2022-03-17T19:04:58.7520036Z * [new branch] gh/pietern/23/orig -> origin/gh/pietern/23/orig 2022-03-17T19:04:58.7521742Z * [new branch] gh/pietern/38/base -> origin/gh/pietern/38/base 2022-03-17T19:04:58.7523789Z * [new branch] gh/pietern/38/orig -> origin/gh/pietern/38/orig 2022-03-17T19:04:58.7525480Z * [new branch] gh/pietern/4/base -> origin/gh/pietern/4/base 2022-03-17T19:04:58.7527516Z * [new branch] gh/pmeier/1/base -> origin/gh/pmeier/1/base 2022-03-17T19:04:58.7528893Z * [new branch] gh/pmeier/1/head -> origin/gh/pmeier/1/head 2022-03-17T19:04:58.7530293Z * [new branch] gh/pmeier/1/orig -> origin/gh/pmeier/1/orig 2022-03-17T19:04:58.7534036Z * [new branch] gh/pmeier/34/base -> origin/gh/pmeier/34/base 2022-03-17T19:04:58.7534657Z * [new branch] gh/pmeier/34/orig -> origin/gh/pmeier/34/orig 2022-03-17T19:04:58.7535411Z * [new branch] gh/pmeier/40/base -> origin/gh/pmeier/40/base 2022-03-17T19:04:58.7536289Z * [new branch] gh/pmeier/40/head -> origin/gh/pmeier/40/head 2022-03-17T19:04:58.7537596Z * [new branch] gh/pmeier/40/orig -> origin/gh/pmeier/40/orig 2022-03-17T19:04:58.7539582Z * [new branch] gh/pmeier/46/base -> origin/gh/pmeier/46/base 2022-03-17T19:04:58.7540835Z * [new branch] gh/pmeier/46/head -> origin/gh/pmeier/46/head 2022-03-17T19:04:58.7542117Z * [new branch] gh/pmeier/46/orig -> origin/gh/pmeier/46/orig 2022-03-17T19:04:58.7544049Z * [new branch] gh/pmeier/47/base -> origin/gh/pmeier/47/base 2022-03-17T19:04:58.7545362Z * [new branch] gh/pmeier/47/head -> origin/gh/pmeier/47/head 2022-03-17T19:04:58.7546783Z * [new branch] gh/pmeier/47/orig -> origin/gh/pmeier/47/orig 2022-03-17T19:04:58.7548510Z * [new branch] gh/pmeier/48/base -> origin/gh/pmeier/48/base 2022-03-17T19:04:58.7549753Z * [new branch] gh/pmeier/48/head -> origin/gh/pmeier/48/head 2022-03-17T19:04:58.7551082Z * [new branch] gh/pmeier/48/orig -> origin/gh/pmeier/48/orig 2022-03-17T19:04:58.7552778Z * [new branch] gh/pmeier/49/base -> origin/gh/pmeier/49/base 2022-03-17T19:04:58.7554080Z * [new branch] gh/pmeier/49/head -> origin/gh/pmeier/49/head 2022-03-17T19:04:58.7555332Z * [new branch] gh/pmeier/49/orig -> origin/gh/pmeier/49/orig 2022-03-17T19:04:58.7557104Z * [new branch] gh/pmeier/53/base -> origin/gh/pmeier/53/base 2022-03-17T19:04:58.7558428Z * [new branch] gh/pmeier/53/head -> origin/gh/pmeier/53/head 2022-03-17T19:04:58.7559768Z * [new branch] gh/pmeier/53/orig -> origin/gh/pmeier/53/orig 2022-03-17T19:04:58.7561460Z * [new branch] gh/pmeier/54/base -> origin/gh/pmeier/54/base 2022-03-17T19:04:58.7562874Z * [new branch] gh/pmeier/54/head -> origin/gh/pmeier/54/head 2022-03-17T19:04:58.7564211Z * [new branch] gh/pmeier/54/orig -> origin/gh/pmeier/54/orig 2022-03-17T19:04:58.7565902Z * [new branch] gh/pmeier/60/base -> origin/gh/pmeier/60/base 2022-03-17T19:04:58.7567147Z * [new branch] gh/pmeier/60/head -> origin/gh/pmeier/60/head 2022-03-17T19:04:58.7568545Z * [new branch] gh/pmeier/60/orig -> origin/gh/pmeier/60/orig 2022-03-17T19:04:58.7570295Z * [new branch] gh/pmeier/61/base -> origin/gh/pmeier/61/base 2022-03-17T19:04:58.7571522Z * [new branch] gh/pmeier/61/head -> origin/gh/pmeier/61/head 2022-03-17T19:04:58.7572829Z * [new branch] gh/pmeier/61/orig -> origin/gh/pmeier/61/orig 2022-03-17T19:04:58.7574631Z * [new branch] gh/pmeier/66/base -> origin/gh/pmeier/66/base 2022-03-17T19:04:58.7575735Z * [new branch] gh/pmeier/66/head -> origin/gh/pmeier/66/head 2022-03-17T19:04:58.7576797Z * [new branch] gh/pmeier/66/orig -> origin/gh/pmeier/66/orig 2022-03-17T19:04:58.7578285Z * [new branch] gh/pmeier/67/base -> origin/gh/pmeier/67/base 2022-03-17T19:04:58.7579666Z * [new branch] gh/pmeier/67/head -> origin/gh/pmeier/67/head 2022-03-17T19:04:58.7581049Z * [new branch] gh/pmeier/67/orig -> origin/gh/pmeier/67/orig 2022-03-17T19:04:58.7582666Z * [new branch] gh/pmeier/70/base -> origin/gh/pmeier/70/base 2022-03-17T19:04:58.7584010Z * [new branch] gh/pmeier/70/head -> origin/gh/pmeier/70/head 2022-03-17T19:04:58.7585259Z * [new branch] gh/pmeier/70/orig -> origin/gh/pmeier/70/orig 2022-03-17T19:04:58.7587265Z * [new branch] gh/pritamdamania87/102/base -> origin/gh/pritamdamania87/102/base 2022-03-17T19:04:58.7588617Z * [new branch] gh/pritamdamania87/102/head -> origin/gh/pritamdamania87/102/head 2022-03-17T19:04:58.7590351Z * [new branch] gh/pritamdamania87/133/base -> origin/gh/pritamdamania87/133/base 2022-03-17T19:04:58.7591861Z * [new branch] gh/pritamdamania87/133/head -> origin/gh/pritamdamania87/133/head 2022-03-17T19:04:58.7593272Z * [new branch] gh/pritamdamania87/134/base -> origin/gh/pritamdamania87/134/base 2022-03-17T19:04:58.7594527Z * [new branch] gh/pritamdamania87/134/head -> origin/gh/pritamdamania87/134/head 2022-03-17T19:04:58.7596004Z * [new branch] gh/pritamdamania87/135/base -> origin/gh/pritamdamania87/135/base 2022-03-17T19:04:58.7597283Z * [new branch] gh/pritamdamania87/135/head -> origin/gh/pritamdamania87/135/head 2022-03-17T19:04:58.7598816Z * [new branch] gh/pritamdamania87/136/base -> origin/gh/pritamdamania87/136/base 2022-03-17T19:04:58.7600110Z * [new branch] gh/pritamdamania87/136/head -> origin/gh/pritamdamania87/136/head 2022-03-17T19:04:58.7601621Z * [new branch] gh/pritamdamania87/137/base -> origin/gh/pritamdamania87/137/base 2022-03-17T19:04:58.7603171Z * [new branch] gh/pritamdamania87/137/head -> origin/gh/pritamdamania87/137/head 2022-03-17T19:04:58.7604661Z * [new branch] gh/pritamdamania87/138/base -> origin/gh/pritamdamania87/138/base 2022-03-17T19:04:58.7605845Z * [new branch] gh/pritamdamania87/138/head -> origin/gh/pritamdamania87/138/head 2022-03-17T19:04:58.7607346Z * [new branch] gh/pritamdamania87/139/base -> origin/gh/pritamdamania87/139/base 2022-03-17T19:04:58.7608621Z * [new branch] gh/pritamdamania87/139/head -> origin/gh/pritamdamania87/139/head 2022-03-17T19:04:58.7610273Z * [new branch] gh/pritamdamania87/157/base -> origin/gh/pritamdamania87/157/base 2022-03-17T19:04:58.7611651Z * [new branch] gh/pritamdamania87/157/head -> origin/gh/pritamdamania87/157/head 2022-03-17T19:04:58.7613028Z * [new branch] gh/pritamdamania87/157/orig -> origin/gh/pritamdamania87/157/orig 2022-03-17T19:04:58.7614917Z * [new branch] gh/pritamdamania87/224/base -> origin/gh/pritamdamania87/224/base 2022-03-17T19:04:58.7616270Z * [new branch] gh/pritamdamania87/224/head -> origin/gh/pritamdamania87/224/head 2022-03-17T19:04:58.7617532Z * [new branch] gh/pritamdamania87/224/orig -> origin/gh/pritamdamania87/224/orig 2022-03-17T19:04:58.7619291Z * [new branch] gh/pritamdamania87/225/base -> origin/gh/pritamdamania87/225/base 2022-03-17T19:04:58.7620629Z * [new branch] gh/pritamdamania87/225/head -> origin/gh/pritamdamania87/225/head 2022-03-17T19:04:58.7622034Z * [new branch] gh/pritamdamania87/225/orig -> origin/gh/pritamdamania87/225/orig 2022-03-17T19:04:58.7623729Z * [new branch] gh/pritamdamania87/39/base -> origin/gh/pritamdamania87/39/base 2022-03-17T19:04:58.7625111Z * [new branch] gh/pritamdamania87/39/head -> origin/gh/pritamdamania87/39/head 2022-03-17T19:04:58.7626344Z * [new branch] gh/pritamdamania87/39/orig -> origin/gh/pritamdamania87/39/orig 2022-03-17T19:04:58.7628293Z * [new branch] gh/priyaramani/5/base -> origin/gh/priyaramani/5/base 2022-03-17T19:04:58.7629665Z * [new branch] gh/priyaramani/5/head -> origin/gh/priyaramani/5/head 2022-03-17T19:04:58.7631136Z * [new branch] gh/priyaramani/6/base -> origin/gh/priyaramani/6/base 2022-03-17T19:04:58.7632511Z * [new branch] gh/priyaramani/6/head -> origin/gh/priyaramani/6/head 2022-03-17T19:04:58.7633989Z * [new branch] gh/priyaramani/7/base -> origin/gh/priyaramani/7/base 2022-03-17T19:04:58.7635216Z * [new branch] gh/priyaramani/7/head -> origin/gh/priyaramani/7/head 2022-03-17T19:04:58.7636881Z * [new branch] gh/priyaramani/9/base -> origin/gh/priyaramani/9/base 2022-03-17T19:04:58.7638263Z * [new branch] gh/priyaramani/9/head -> origin/gh/priyaramani/9/head 2022-03-17T19:04:58.7639567Z * [new branch] gh/priyaramani/9/orig -> origin/gh/priyaramani/9/orig 2022-03-17T19:04:58.7641732Z * [new branch] gh/qihan/1/base -> origin/gh/qihan/1/base 2022-03-17T19:04:58.7643196Z * [new branch] gh/qihan/1/head -> origin/gh/qihan/1/head 2022-03-17T19:04:58.7644472Z * [new branch] gh/qihan/1/orig -> origin/gh/qihan/1/orig 2022-03-17T19:04:58.7647059Z * [new branch] gh/raghuraman_k/1/base -> origin/gh/raghuraman_k/1/base 2022-03-17T19:04:58.7648010Z * [new branch] gh/raghuraman_k/1/head -> origin/gh/raghuraman_k/1/head 2022-03-17T19:04:58.7650168Z * [new branch] gh/raghuraman_k/11/base -> origin/gh/raghuraman_k/11/base 2022-03-17T19:04:58.7651293Z * [new branch] gh/raghuraman_k/11/head -> origin/gh/raghuraman_k/11/head 2022-03-17T19:04:58.7652875Z * [new branch] gh/raghuraman_k/11/orig -> origin/gh/raghuraman_k/11/orig 2022-03-17T19:04:58.7654596Z * [new branch] gh/raghuraman_k/13/base -> origin/gh/raghuraman_k/13/base 2022-03-17T19:04:58.7655774Z * [new branch] gh/raghuraman_k/13/head -> origin/gh/raghuraman_k/13/head 2022-03-17T19:04:58.7657152Z * [new branch] gh/raghuraman_k/13/orig -> origin/gh/raghuraman_k/13/orig 2022-03-17T19:04:58.7659100Z * [new branch] gh/raghuraman_k/15/base -> origin/gh/raghuraman_k/15/base 2022-03-17T19:04:58.7660227Z * [new branch] gh/raghuraman_k/15/head -> origin/gh/raghuraman_k/15/head 2022-03-17T19:04:58.7662023Z * [new branch] gh/raghuraman_k/15/orig -> origin/gh/raghuraman_k/15/orig 2022-03-17T19:04:58.7663728Z * [new branch] gh/raghuraman_k/17/base -> origin/gh/raghuraman_k/17/base 2022-03-17T19:04:58.7664763Z * [new branch] gh/raghuraman_k/17/head -> origin/gh/raghuraman_k/17/head 2022-03-17T19:04:58.7666430Z * [new branch] gh/raghuraman_k/17/orig -> origin/gh/raghuraman_k/17/orig 2022-03-17T19:04:58.7668098Z * [new branch] gh/raghuraman_k/19/base -> origin/gh/raghuraman_k/19/base 2022-03-17T19:04:58.7669322Z * [new branch] gh/raghuraman_k/19/head -> origin/gh/raghuraman_k/19/head 2022-03-17T19:04:58.7670752Z * [new branch] gh/raghuraman_k/19/orig -> origin/gh/raghuraman_k/19/orig 2022-03-17T19:04:58.7672257Z * [new branch] gh/raghuraman_k/2/base -> origin/gh/raghuraman_k/2/base 2022-03-17T19:04:58.7673358Z * [new branch] gh/raghuraman_k/2/head -> origin/gh/raghuraman_k/2/head 2022-03-17T19:04:58.7675251Z * [new branch] gh/raghuraman_k/20/base -> origin/gh/raghuraman_k/20/base 2022-03-17T19:04:58.7676596Z * [new branch] gh/raghuraman_k/20/head -> origin/gh/raghuraman_k/20/head 2022-03-17T19:04:58.7677676Z * [new branch] gh/raghuraman_k/20/orig -> origin/gh/raghuraman_k/20/orig 2022-03-17T19:04:58.7679535Z * [new branch] gh/raghuraman_k/21/base -> origin/gh/raghuraman_k/21/base 2022-03-17T19:04:58.7680701Z * [new branch] gh/raghuraman_k/21/head -> origin/gh/raghuraman_k/21/head 2022-03-17T19:04:58.7682278Z * [new branch] gh/raghuraman_k/21/orig -> origin/gh/raghuraman_k/21/orig 2022-03-17T19:04:58.7684640Z * [new branch] gh/raghuraman_k/4/base -> origin/gh/raghuraman_k/4/base 2022-03-17T19:04:58.7685841Z * [new branch] gh/raghuraman_k/4/head -> origin/gh/raghuraman_k/4/head 2022-03-17T19:04:58.7687285Z * [new branch] gh/raghuraman_k/4/orig -> origin/gh/raghuraman_k/4/orig 2022-03-17T19:04:58.7689093Z * [new branch] gh/raghuraman_k/5/base -> origin/gh/raghuraman_k/5/base 2022-03-17T19:04:58.7690478Z * [new branch] gh/raghuraman_k/5/head -> origin/gh/raghuraman_k/5/head 2022-03-17T19:04:58.7691615Z * [new branch] gh/raghuraman_k/5/orig -> origin/gh/raghuraman_k/5/orig 2022-03-17T19:04:58.7693224Z * [new branch] gh/raghuraman_k/6/base -> origin/gh/raghuraman_k/6/base 2022-03-17T19:04:58.7694547Z * [new branch] gh/raghuraman_k/6/head -> origin/gh/raghuraman_k/6/head 2022-03-17T19:04:58.7695693Z * [new branch] gh/raghuraman_k/6/orig -> origin/gh/raghuraman_k/6/orig 2022-03-17T19:04:58.7697337Z * [new branch] gh/raghuraman_k/7/base -> origin/gh/raghuraman_k/7/base 2022-03-17T19:04:58.7698408Z * [new branch] gh/raghuraman_k/7/head -> origin/gh/raghuraman_k/7/head 2022-03-17T19:04:58.7699818Z * [new branch] gh/raghuraman_k/7/orig -> origin/gh/raghuraman_k/7/orig 2022-03-17T19:04:58.7701613Z * [new branch] gh/raghuraman_k/9/base -> origin/gh/raghuraman_k/9/base 2022-03-17T19:04:58.7702961Z * [new branch] gh/raghuraman_k/9/head -> origin/gh/raghuraman_k/9/head 2022-03-17T19:04:58.7704325Z * [new branch] gh/raghuraman_k/9/orig -> origin/gh/raghuraman_k/9/orig 2022-03-17T19:04:58.7706323Z * [new branch] gh/raghuramank100/10/base -> origin/gh/raghuramank100/10/base 2022-03-17T19:04:58.7707494Z * [new branch] gh/raghuramank100/10/head -> origin/gh/raghuramank100/10/head 2022-03-17T19:04:58.7709108Z * [new branch] gh/raghuramank100/10/orig -> origin/gh/raghuramank100/10/orig 2022-03-17T19:04:58.7710549Z * [new branch] gh/raghuramank100/11/base -> origin/gh/raghuramank100/11/base 2022-03-17T19:04:58.7711947Z * [new branch] gh/raghuramank100/11/head -> origin/gh/raghuramank100/11/head 2022-03-17T19:04:58.7713111Z * [new branch] gh/raghuramank100/11/orig -> origin/gh/raghuramank100/11/orig 2022-03-17T19:04:58.7714910Z * [new branch] gh/raghuramank100/12/base -> origin/gh/raghuramank100/12/base 2022-03-17T19:04:58.7716467Z * [new branch] gh/raghuramank100/12/head -> origin/gh/raghuramank100/12/head 2022-03-17T19:04:58.7717599Z * [new branch] gh/raghuramank100/12/orig -> origin/gh/raghuramank100/12/orig 2022-03-17T19:04:58.7719242Z * [new branch] gh/raghuramank100/16/base -> origin/gh/raghuramank100/16/base 2022-03-17T19:04:58.7720431Z * [new branch] gh/raghuramank100/16/head -> origin/gh/raghuramank100/16/head 2022-03-17T19:04:58.7722115Z * [new branch] gh/raghuramank100/19/base -> origin/gh/raghuramank100/19/base 2022-03-17T19:04:58.7723644Z * [new branch] gh/raghuramank100/19/head -> origin/gh/raghuramank100/19/head 2022-03-17T19:04:58.7724892Z * [new branch] gh/raghuramank100/19/orig -> origin/gh/raghuramank100/19/orig 2022-03-17T19:04:58.7726697Z * [new branch] gh/raghuramank100/22/base -> origin/gh/raghuramank100/22/base 2022-03-17T19:04:58.7727886Z * [new branch] gh/raghuramank100/22/head -> origin/gh/raghuramank100/22/head 2022-03-17T19:04:58.7729393Z * [new branch] gh/raghuramank100/22/orig -> origin/gh/raghuramank100/22/orig 2022-03-17T19:04:58.7730990Z * [new branch] gh/raghuramank100/26/base -> origin/gh/raghuramank100/26/base 2022-03-17T19:04:58.7732386Z * [new branch] gh/raghuramank100/26/head -> origin/gh/raghuramank100/26/head 2022-03-17T19:04:58.7733519Z * [new branch] gh/raghuramank100/26/orig -> origin/gh/raghuramank100/26/orig 2022-03-17T19:04:58.7735383Z * [new branch] gh/raghuramank100/28/base -> origin/gh/raghuramank100/28/base 2022-03-17T19:04:58.7736596Z * [new branch] gh/raghuramank100/28/head -> origin/gh/raghuramank100/28/head 2022-03-17T19:04:58.7738162Z * [new branch] gh/raghuramank100/28/orig -> origin/gh/raghuramank100/28/orig 2022-03-17T19:04:58.7739805Z * [new branch] gh/raghuramank100/30/base -> origin/gh/raghuramank100/30/base 2022-03-17T19:04:58.7741005Z * [new branch] gh/raghuramank100/30/head -> origin/gh/raghuramank100/30/head 2022-03-17T19:04:58.7742500Z * [new branch] gh/raghuramank100/30/orig -> origin/gh/raghuramank100/30/orig 2022-03-17T19:04:58.7744211Z * [new branch] gh/raghuramank100/37/base -> origin/gh/raghuramank100/37/base 2022-03-17T19:04:58.7745804Z * [new branch] gh/raghuramank100/37/head -> origin/gh/raghuramank100/37/head 2022-03-17T19:04:58.7746935Z * [new branch] gh/raghuramank100/37/orig -> origin/gh/raghuramank100/37/orig 2022-03-17T19:04:58.7748751Z * [new branch] gh/raghuramank100/39/base -> origin/gh/raghuramank100/39/base 2022-03-17T19:04:58.7749956Z * [new branch] gh/raghuramank100/39/head -> origin/gh/raghuramank100/39/head 2022-03-17T19:04:58.7751469Z * [new branch] gh/raghuramank100/39/orig -> origin/gh/raghuramank100/39/orig 2022-03-17T19:04:58.7753167Z * [new branch] gh/raghuramank100/4/base -> origin/gh/raghuramank100/4/base 2022-03-17T19:04:58.7754709Z * [new branch] gh/raghuramank100/4/head -> origin/gh/raghuramank100/4/head 2022-03-17T19:04:58.7755761Z * [new branch] gh/raghuramank100/4/orig -> origin/gh/raghuramank100/4/orig 2022-03-17T19:04:58.7757563Z * [new branch] gh/raghuramank100/40/base -> origin/gh/raghuramank100/40/base 2022-03-17T19:04:58.7758915Z * [new branch] gh/raghuramank100/40/head -> origin/gh/raghuramank100/40/head 2022-03-17T19:04:58.7760343Z * [new branch] gh/raghuramank100/40/orig -> origin/gh/raghuramank100/40/orig 2022-03-17T19:04:58.7761942Z * [new branch] gh/raghuramank100/41/base -> origin/gh/raghuramank100/41/base 2022-03-17T19:04:58.7764272Z * [new branch] gh/raghuramank100/41/head -> origin/gh/raghuramank100/41/head 2022-03-17T19:04:58.7765392Z * [new branch] gh/raghuramank100/41/orig -> origin/gh/raghuramank100/41/orig 2022-03-17T19:04:58.7766989Z * [new branch] gh/raghuramank100/42/base -> origin/gh/raghuramank100/42/base 2022-03-17T19:04:58.7768128Z * [new branch] gh/raghuramank100/42/head -> origin/gh/raghuramank100/42/head 2022-03-17T19:04:58.7769520Z * [new branch] gh/raghuramank100/42/orig -> origin/gh/raghuramank100/42/orig 2022-03-17T19:04:58.7771232Z * [new branch] gh/raghuramank100/43/base -> origin/gh/raghuramank100/43/base 2022-03-17T19:04:58.7772374Z * [new branch] gh/raghuramank100/43/head -> origin/gh/raghuramank100/43/head 2022-03-17T19:04:58.7773820Z * [new branch] gh/raghuramank100/43/orig -> origin/gh/raghuramank100/43/orig 2022-03-17T19:04:58.7775532Z * [new branch] gh/raghuramank100/46/base -> origin/gh/raghuramank100/46/base 2022-03-17T19:04:58.7776917Z * [new branch] gh/raghuramank100/46/head -> origin/gh/raghuramank100/46/head 2022-03-17T19:04:58.7778118Z * [new branch] gh/raghuramank100/46/orig -> origin/gh/raghuramank100/46/orig 2022-03-17T19:04:58.7780201Z * [new branch] gh/raghuramank100/47/base -> origin/gh/raghuramank100/47/base 2022-03-17T19:04:58.7781574Z * [new branch] gh/raghuramank100/47/head -> origin/gh/raghuramank100/47/head 2022-03-17T19:04:58.7782948Z * [new branch] gh/raghuramank100/47/orig -> origin/gh/raghuramank100/47/orig 2022-03-17T19:04:58.7784766Z * [new branch] gh/raghuramank100/48/base -> origin/gh/raghuramank100/48/base 2022-03-17T19:04:58.7785989Z * [new branch] gh/raghuramank100/48/head -> origin/gh/raghuramank100/48/head 2022-03-17T19:04:58.7787456Z * [new branch] gh/raghuramank100/48/orig -> origin/gh/raghuramank100/48/orig 2022-03-17T19:04:58.7789162Z * [new branch] gh/raghuramank100/49/base -> origin/gh/raghuramank100/49/base 2022-03-17T19:04:58.7790548Z * [new branch] gh/raghuramank100/49/head -> origin/gh/raghuramank100/49/head 2022-03-17T19:04:58.7791824Z * [new branch] gh/raghuramank100/49/orig -> origin/gh/raghuramank100/49/orig 2022-03-17T19:04:58.7793852Z * [new branch] gh/raghuramank100/5/base -> origin/gh/raghuramank100/5/base 2022-03-17T19:04:58.7795059Z * [new branch] gh/raghuramank100/5/head -> origin/gh/raghuramank100/5/head 2022-03-17T19:04:58.7796619Z * [new branch] gh/raghuramank100/5/orig -> origin/gh/raghuramank100/5/orig 2022-03-17T19:04:58.7798301Z * [new branch] gh/raghuramank100/50/base -> origin/gh/raghuramank100/50/base 2022-03-17T19:04:58.7799579Z * [new branch] gh/raghuramank100/50/head -> origin/gh/raghuramank100/50/head 2022-03-17T19:04:58.7800998Z * [new branch] gh/raghuramank100/50/orig -> origin/gh/raghuramank100/50/orig 2022-03-17T19:04:58.7802725Z * [new branch] gh/raghuramank100/6/base -> origin/gh/raghuramank100/6/base 2022-03-17T19:04:58.7803986Z * [new branch] gh/raghuramank100/6/head -> origin/gh/raghuramank100/6/head 2022-03-17T19:04:58.7805222Z * [new branch] gh/raghuramank100/6/orig -> origin/gh/raghuramank100/6/orig 2022-03-17T19:04:58.7807059Z * [new branch] gh/raghuramank100/7/base -> origin/gh/raghuramank100/7/base 2022-03-17T19:04:58.7808352Z * [new branch] gh/raghuramank100/7/head -> origin/gh/raghuramank100/7/head 2022-03-17T19:04:58.7809811Z * [new branch] gh/raghuramank100/7/orig -> origin/gh/raghuramank100/7/orig 2022-03-17T19:04:58.7811518Z * [new branch] gh/raghuramank100/8/base -> origin/gh/raghuramank100/8/base 2022-03-17T19:04:58.7812814Z * [new branch] gh/raghuramank100/8/head -> origin/gh/raghuramank100/8/head 2022-03-17T19:04:58.7814175Z * [new branch] gh/raghuramank100/8/orig -> origin/gh/raghuramank100/8/orig 2022-03-17T19:04:58.7816408Z * [new branch] gh/robieta/11/base -> origin/gh/robieta/11/base 2022-03-17T19:04:58.7817716Z * [new branch] gh/robieta/11/head -> origin/gh/robieta/11/head 2022-03-17T19:04:58.7819305Z * [new branch] gh/robieta/12/base -> origin/gh/robieta/12/base 2022-03-17T19:04:58.7821174Z * [new branch] gh/robieta/13/base -> origin/gh/robieta/13/base 2022-03-17T19:04:58.7822749Z * [new branch] gh/robieta/22/base -> origin/gh/robieta/22/base 2022-03-17T19:04:58.7824259Z * [new branch] gh/robieta/22/head -> origin/gh/robieta/22/head 2022-03-17T19:04:58.7825449Z * [new branch] gh/robieta/22/orig -> origin/gh/robieta/22/orig 2022-03-17T19:04:58.7827430Z * [new branch] gh/robieta/24/base -> origin/gh/robieta/24/base 2022-03-17T19:04:58.7828939Z * [new branch] gh/robieta/24/head -> origin/gh/robieta/24/head 2022-03-17T19:04:58.7830129Z * [new branch] gh/robieta/24/orig -> origin/gh/robieta/24/orig 2022-03-17T19:04:58.7831892Z * [new branch] gh/robieta/25/base -> origin/gh/robieta/25/base 2022-03-17T19:04:58.7833406Z * [new branch] gh/robieta/25/head -> origin/gh/robieta/25/head 2022-03-17T19:04:58.7834668Z * [new branch] gh/robieta/25/orig -> origin/gh/robieta/25/orig 2022-03-17T19:04:58.7836507Z * [new branch] gh/robieta/26/base -> origin/gh/robieta/26/base 2022-03-17T19:04:58.7837901Z * [new branch] gh/robieta/26/head -> origin/gh/robieta/26/head 2022-03-17T19:04:58.7839166Z * [new branch] gh/robieta/26/orig -> origin/gh/robieta/26/orig 2022-03-17T19:04:58.7841211Z * [new branch] gh/robieta/30/base -> origin/gh/robieta/30/base 2022-03-17T19:04:58.7842860Z * [new branch] gh/robieta/30/head -> origin/gh/robieta/30/head 2022-03-17T19:04:58.7844194Z * [new branch] gh/robieta/30/orig -> origin/gh/robieta/30/orig 2022-03-17T19:04:58.7845997Z * [new branch] gh/robieta/31/base -> origin/gh/robieta/31/base 2022-03-17T19:04:58.7847302Z * [new branch] gh/robieta/31/head -> origin/gh/robieta/31/head 2022-03-17T19:04:58.7848580Z * [new branch] gh/robieta/31/orig -> origin/gh/robieta/31/orig 2022-03-17T19:04:58.7850790Z * [new branch] gh/robieta/32/base -> origin/gh/robieta/32/base 2022-03-17T19:04:58.7851889Z * [new branch] gh/robieta/32/head -> origin/gh/robieta/32/head 2022-03-17T19:04:58.7853171Z * [new branch] gh/robieta/32/orig -> origin/gh/robieta/32/orig 2022-03-17T19:04:58.7854946Z * [new branch] gh/robieta/33/base -> origin/gh/robieta/33/base 2022-03-17T19:04:58.7856232Z * [new branch] gh/robieta/33/head -> origin/gh/robieta/33/head 2022-03-17T19:04:58.7857538Z * [new branch] gh/robieta/33/orig -> origin/gh/robieta/33/orig 2022-03-17T19:04:58.7859392Z * [new branch] gh/robieta/34/base -> origin/gh/robieta/34/base 2022-03-17T19:04:58.7860668Z * [new branch] gh/robieta/34/head -> origin/gh/robieta/34/head 2022-03-17T19:04:58.7861931Z * [new branch] gh/robieta/34/orig -> origin/gh/robieta/34/orig 2022-03-17T19:04:58.7863847Z * [new branch] gh/robieta/35/base -> origin/gh/robieta/35/base 2022-03-17T19:04:58.7865264Z * [new branch] gh/robieta/35/head -> origin/gh/robieta/35/head 2022-03-17T19:04:58.7866658Z * [new branch] gh/robieta/35/orig -> origin/gh/robieta/35/orig 2022-03-17T19:04:58.7868471Z * [new branch] gh/robieta/36/base -> origin/gh/robieta/36/base 2022-03-17T19:04:58.7869747Z * [new branch] gh/robieta/36/head -> origin/gh/robieta/36/head 2022-03-17T19:04:58.7871040Z * [new branch] gh/robieta/36/orig -> origin/gh/robieta/36/orig 2022-03-17T19:04:58.7872706Z * [new branch] gh/robieta/37/base -> origin/gh/robieta/37/base 2022-03-17T19:04:58.7874158Z * [new branch] gh/robieta/37/head -> origin/gh/robieta/37/head 2022-03-17T19:04:58.7875354Z * [new branch] gh/robieta/37/orig -> origin/gh/robieta/37/orig 2022-03-17T19:04:58.7876712Z * [new branch] gh/robieta/benchmark_utils -> origin/gh/robieta/benchmark_utils 2022-03-17T19:04:58.7877987Z * [new branch] gh/robieta/schedule_functor -> origin/gh/robieta/schedule_functor 2022-03-17T19:04:58.7879986Z * [new branch] gh/rohan-varma/1/base -> origin/gh/rohan-varma/1/base 2022-03-17T19:04:58.7881245Z * [new branch] gh/rohan-varma/1/orig -> origin/gh/rohan-varma/1/orig 2022-03-17T19:04:58.7883435Z * [new branch] gh/rohan-varma/160/base -> origin/gh/rohan-varma/160/base 2022-03-17T19:04:58.7884693Z * [new branch] gh/rohan-varma/160/head -> origin/gh/rohan-varma/160/head 2022-03-17T19:04:58.7886444Z * [new branch] gh/rohan-varma/191/base -> origin/gh/rohan-varma/191/base 2022-03-17T19:04:58.7887808Z * [new branch] gh/rohan-varma/191/head -> origin/gh/rohan-varma/191/head 2022-03-17T19:04:58.7889101Z * [new branch] gh/rohan-varma/191/orig -> origin/gh/rohan-varma/191/orig 2022-03-17T19:04:58.7890807Z * [new branch] gh/rohan-varma/256/base -> origin/gh/rohan-varma/256/base 2022-03-17T19:04:58.7892135Z * [new branch] gh/rohan-varma/256/head -> origin/gh/rohan-varma/256/head 2022-03-17T19:04:58.7893936Z * [new branch] gh/rohan-varma/284/base -> origin/gh/rohan-varma/284/base 2022-03-17T19:04:58.7895474Z * [new branch] gh/rohan-varma/284/head -> origin/gh/rohan-varma/284/head 2022-03-17T19:04:58.7896776Z * [new branch] gh/rohan-varma/284/orig -> origin/gh/rohan-varma/284/orig 2022-03-17T19:04:58.7898587Z * [new branch] gh/rohan-varma/34/base -> origin/gh/rohan-varma/34/base 2022-03-17T19:04:58.7900074Z * [new branch] gh/rohan-varma/34/head -> origin/gh/rohan-varma/34/head 2022-03-17T19:04:58.7901375Z * [new branch] gh/rohan-varma/34/orig -> origin/gh/rohan-varma/34/orig 2022-03-17T19:04:58.7903201Z * [new branch] gh/rohan-varma/378/base -> origin/gh/rohan-varma/378/base 2022-03-17T19:04:58.7904727Z * [new branch] gh/rohan-varma/378/head -> origin/gh/rohan-varma/378/head 2022-03-17T19:04:58.7905949Z * [new branch] gh/rohan-varma/378/orig -> origin/gh/rohan-varma/378/orig 2022-03-17T19:04:58.7907371Z * [new branch] gh/rohan-varma/379/base -> origin/gh/rohan-varma/379/base 2022-03-17T19:04:58.7908692Z * [new branch] gh/rohan-varma/379/head -> origin/gh/rohan-varma/379/head 2022-03-17T19:04:58.7910101Z * [new branch] gh/rohan-varma/379/orig -> origin/gh/rohan-varma/379/orig 2022-03-17T19:04:58.7911919Z * [new branch] gh/rohan-varma/411/base -> origin/gh/rohan-varma/411/base 2022-03-17T19:04:58.7913210Z * [new branch] gh/rohan-varma/411/head -> origin/gh/rohan-varma/411/head 2022-03-17T19:04:58.7914575Z * [new branch] gh/rohan-varma/411/orig -> origin/gh/rohan-varma/411/orig 2022-03-17T19:04:58.7916451Z * [new branch] gh/rohan-varma/438/base -> origin/gh/rohan-varma/438/base 2022-03-17T19:04:58.7918035Z * [new branch] gh/rohan-varma/438/head -> origin/gh/rohan-varma/438/head 2022-03-17T19:04:58.7919491Z * [new branch] gh/rohan-varma/438/orig -> origin/gh/rohan-varma/438/orig 2022-03-17T19:04:58.7921268Z * [new branch] gh/rohan-varma/466/base -> origin/gh/rohan-varma/466/base 2022-03-17T19:04:58.7922857Z * [new branch] gh/rohan-varma/466/head -> origin/gh/rohan-varma/466/head 2022-03-17T19:04:58.7925239Z * [new branch] gh/rohan-varma/466/orig -> origin/gh/rohan-varma/466/orig 2022-03-17T19:04:58.7926603Z * [new branch] gh/rohan-varma/472/base -> origin/gh/rohan-varma/472/base 2022-03-17T19:04:58.7927565Z * [new branch] gh/rohan-varma/472/head -> origin/gh/rohan-varma/472/head 2022-03-17T19:04:58.7929606Z * [new branch] gh/rohan-varma/472/orig -> origin/gh/rohan-varma/472/orig 2022-03-17T19:04:58.7931582Z * [new branch] gh/rohan-varma/491/base -> origin/gh/rohan-varma/491/base 2022-03-17T19:04:58.7932349Z * [new branch] gh/rohan-varma/491/head -> origin/gh/rohan-varma/491/head 2022-03-17T19:04:58.7933923Z * [new branch] gh/rohan-varma/491/orig -> origin/gh/rohan-varma/491/orig 2022-03-17T19:04:58.7935603Z * [new branch] gh/rohan-varma/506/base -> origin/gh/rohan-varma/506/base 2022-03-17T19:04:58.7937177Z * [new branch] gh/rohan-varma/506/head -> origin/gh/rohan-varma/506/head 2022-03-17T19:04:58.7938365Z * [new branch] gh/rohan-varma/506/orig -> origin/gh/rohan-varma/506/orig 2022-03-17T19:04:58.7940057Z * [new branch] gh/rohan-varma/516/base -> origin/gh/rohan-varma/516/base 2022-03-17T19:04:58.7941179Z * [new branch] gh/rohan-varma/516/head -> origin/gh/rohan-varma/516/head 2022-03-17T19:04:58.7942966Z * [new branch] gh/rohan-varma/516/orig -> origin/gh/rohan-varma/516/orig 2022-03-17T19:04:58.7944576Z * [new branch] gh/rohan-varma/518/base -> origin/gh/rohan-varma/518/base 2022-03-17T19:04:58.7945661Z * [new branch] gh/rohan-varma/518/head -> origin/gh/rohan-varma/518/head 2022-03-17T19:04:58.7947229Z * [new branch] gh/rohan-varma/518/orig -> origin/gh/rohan-varma/518/orig 2022-03-17T19:04:58.7948941Z * [new branch] gh/rohan-varma/519/base -> origin/gh/rohan-varma/519/base 2022-03-17T19:04:58.7950053Z * [new branch] gh/rohan-varma/519/head -> origin/gh/rohan-varma/519/head 2022-03-17T19:04:58.7951602Z * [new branch] gh/rohan-varma/519/orig -> origin/gh/rohan-varma/519/orig 2022-03-17T19:04:58.7953307Z * [new branch] gh/rohan-varma/520/base -> origin/gh/rohan-varma/520/base 2022-03-17T19:04:58.7954420Z * [new branch] gh/rohan-varma/520/head -> origin/gh/rohan-varma/520/head 2022-03-17T19:04:58.7956168Z * [new branch] gh/rohan-varma/520/orig -> origin/gh/rohan-varma/520/orig 2022-03-17T19:04:58.7957692Z * [new branch] gh/rohan-varma/521/base -> origin/gh/rohan-varma/521/base 2022-03-17T19:04:58.7958726Z * [new branch] gh/rohan-varma/521/head -> origin/gh/rohan-varma/521/head 2022-03-17T19:04:58.7960381Z * [new branch] gh/rohan-varma/521/orig -> origin/gh/rohan-varma/521/orig 2022-03-17T19:04:58.7962227Z * [new branch] gh/rohan-varma/522/base -> origin/gh/rohan-varma/522/base 2022-03-17T19:04:58.7963713Z * [new branch] gh/rohan-varma/522/head -> origin/gh/rohan-varma/522/head 2022-03-17T19:04:58.7964876Z * [new branch] gh/rohan-varma/522/orig -> origin/gh/rohan-varma/522/orig 2022-03-17T19:04:58.7966509Z * [new branch] gh/rohan-varma/523/base -> origin/gh/rohan-varma/523/base 2022-03-17T19:04:58.7967687Z * [new branch] gh/rohan-varma/523/head -> origin/gh/rohan-varma/523/head 2022-03-17T19:04:58.7969233Z * [new branch] gh/rohan-varma/523/orig -> origin/gh/rohan-varma/523/orig 2022-03-17T19:04:58.7970560Z * [new branch] gh/rohan-varma/524/base -> origin/gh/rohan-varma/524/base 2022-03-17T19:04:58.7972063Z * [new branch] gh/rohan-varma/524/head -> origin/gh/rohan-varma/524/head 2022-03-17T19:04:58.7973158Z * [new branch] gh/rohan-varma/524/orig -> origin/gh/rohan-varma/524/orig 2022-03-17T19:04:58.7975029Z * [new branch] gh/rohan-varma/9/base -> origin/gh/rohan-varma/9/base 2022-03-17T19:04:58.7976191Z * [new branch] gh/rohan-varma/9/head -> origin/gh/rohan-varma/9/head 2022-03-17T19:04:58.7977922Z * [new branch] gh/rohan-varma/9/orig -> origin/gh/rohan-varma/9/orig 2022-03-17T19:04:58.7979629Z * [new branch] gh/rsemenov/2/base -> origin/gh/rsemenov/2/base 2022-03-17T19:04:58.7980744Z * [new branch] gh/rsemenov/2/head -> origin/gh/rsemenov/2/head 2022-03-17T19:04:58.7982330Z * [new branch] gh/rsemenov/2/orig -> origin/gh/rsemenov/2/orig 2022-03-17T19:04:58.7984404Z * [new branch] gh/sahancpal@gmail.com/1/base -> origin/gh/sahancpal@gmail.com/1/base 2022-03-17T19:04:58.7985504Z * [new branch] gh/sahancpal@gmail.com/1/orig -> origin/gh/sahancpal@gmail.com/1/orig 2022-03-17T19:04:58.7987421Z * [new branch] gh/sahancpal@gmail.com/2/base -> origin/gh/sahancpal@gmail.com/2/base 2022-03-17T19:04:58.7988635Z * [new branch] gh/sahancpal@gmail.com/2/orig -> origin/gh/sahancpal@gmail.com/2/orig 2022-03-17T19:04:58.7990491Z * [new branch] gh/sahancpal@gmail.com/3/base -> origin/gh/sahancpal@gmail.com/3/base 2022-03-17T19:04:58.7991670Z * [new branch] gh/sahancpal@gmail.com/3/orig -> origin/gh/sahancpal@gmail.com/3/orig 2022-03-17T19:04:58.7993638Z * [new branch] gh/salexspb/1/base -> origin/gh/salexspb/1/base 2022-03-17T19:04:58.7995113Z * [new branch] gh/salexspb/1/head -> origin/gh/salexspb/1/head 2022-03-17T19:04:58.7996591Z * [new branch] gh/salexspb/2/base -> origin/gh/salexspb/2/base 2022-03-17T19:04:58.7997610Z * [new branch] gh/salexspb/2/head -> origin/gh/salexspb/2/head 2022-03-17T19:04:58.7999843Z * [new branch] gh/salilsdesai/14/base -> origin/gh/salilsdesai/14/base 2022-03-17T19:04:58.8000961Z * [new branch] gh/salilsdesai/14/head -> origin/gh/salilsdesai/14/head 2022-03-17T19:04:58.8002599Z * [new branch] gh/salilsdesai/14/orig -> origin/gh/salilsdesai/14/orig 2022-03-17T19:04:58.8004611Z * [new branch] gh/salilsdesai/15/base -> origin/gh/salilsdesai/15/base 2022-03-17T19:04:58.8005754Z * [new branch] gh/salilsdesai/15/head -> origin/gh/salilsdesai/15/head 2022-03-17T19:04:58.8007226Z * [new branch] gh/salilsdesai/15/orig -> origin/gh/salilsdesai/15/orig 2022-03-17T19:04:58.8009085Z * [new branch] gh/salilsdesai/16/base -> origin/gh/salilsdesai/16/base 2022-03-17T19:04:58.8010132Z * [new branch] gh/salilsdesai/16/head -> origin/gh/salilsdesai/16/head 2022-03-17T19:04:58.8011626Z * [new branch] gh/salilsdesai/16/orig -> origin/gh/salilsdesai/16/orig 2022-03-17T19:04:58.8013390Z * [new branch] gh/salilsdesai/17/base -> origin/gh/salilsdesai/17/base 2022-03-17T19:04:58.8014562Z * [new branch] gh/salilsdesai/17/head -> origin/gh/salilsdesai/17/head 2022-03-17T19:04:58.8016031Z * [new branch] gh/salilsdesai/17/orig -> origin/gh/salilsdesai/17/orig 2022-03-17T19:04:58.8017673Z * [new branch] gh/salilsdesai/5/base -> origin/gh/salilsdesai/5/base 2022-03-17T19:04:58.8019132Z * [new branch] gh/salilsdesai/5/head -> origin/gh/salilsdesai/5/head 2022-03-17T19:04:58.8020491Z * [new branch] gh/salilsdesai/5/orig -> origin/gh/salilsdesai/5/orig 2022-03-17T19:04:58.8021965Z * [new branch] gh/salilsdesai/6/base -> origin/gh/salilsdesai/6/base 2022-03-17T19:04:58.8023242Z * [new branch] gh/salilsdesai/6/head -> origin/gh/salilsdesai/6/head 2022-03-17T19:04:58.8024774Z * [new branch] gh/salilsdesai/6/orig -> origin/gh/salilsdesai/6/orig 2022-03-17T19:04:58.8026720Z * [new branch] gh/samdow/2/base -> origin/gh/samdow/2/base 2022-03-17T19:04:58.8027824Z * [new branch] gh/samdow/2/head -> origin/gh/samdow/2/head 2022-03-17T19:04:58.8029245Z * [new branch] gh/samdow/2/orig -> origin/gh/samdow/2/orig 2022-03-17T19:04:58.8031030Z * [new branch] gh/samdow/4/base -> origin/gh/samdow/4/base 2022-03-17T19:04:58.8032367Z * [new branch] gh/samdow/4/head -> origin/gh/samdow/4/head 2022-03-17T19:04:58.8033665Z * [new branch] gh/samdow/4/orig -> origin/gh/samdow/4/orig 2022-03-17T19:04:58.8035369Z * [new branch] gh/samdow/5/base -> origin/gh/samdow/5/base 2022-03-17T19:04:58.8036414Z * [new branch] gh/samdow/5/head -> origin/gh/samdow/5/head 2022-03-17T19:04:58.8037824Z * [new branch] gh/samdow/5/orig -> origin/gh/samdow/5/orig 2022-03-17T19:04:58.8039371Z * [new branch] gh/samdow/6/base -> origin/gh/samdow/6/base 2022-03-17T19:04:58.8040677Z * [new branch] gh/samdow/6/head -> origin/gh/samdow/6/head 2022-03-17T19:04:58.8041773Z * [new branch] gh/samdow/6/orig -> origin/gh/samdow/6/orig 2022-03-17T19:04:58.8044434Z * [new branch] gh/seemethere/103/base -> origin/gh/seemethere/103/base 2022-03-17T19:04:58.8045555Z * [new branch] gh/seemethere/103/head -> origin/gh/seemethere/103/head 2022-03-17T19:04:58.8046787Z * [new branch] gh/seemethere/103/orig -> origin/gh/seemethere/103/orig 2022-03-17T19:04:58.8048561Z * [new branch] gh/seemethere/110/base -> origin/gh/seemethere/110/base 2022-03-17T19:04:58.8049813Z * [new branch] gh/seemethere/110/orig -> origin/gh/seemethere/110/orig 2022-03-17T19:04:58.8051529Z * [new branch] gh/seemethere/14/base -> origin/gh/seemethere/14/base 2022-03-17T19:04:58.8052826Z * [new branch] gh/seemethere/14/orig -> origin/gh/seemethere/14/orig 2022-03-17T19:04:58.8054579Z * [new branch] gh/seemethere/156/base -> origin/gh/seemethere/156/base 2022-03-17T19:04:58.8055892Z * [new branch] gh/seemethere/156/head -> origin/gh/seemethere/156/head 2022-03-17T19:04:58.8057347Z * [new branch] gh/seemethere/156/orig -> origin/gh/seemethere/156/orig 2022-03-17T19:04:58.8059196Z * [new branch] gh/seemethere/159/base -> origin/gh/seemethere/159/base 2022-03-17T19:04:58.8060537Z * [new branch] gh/seemethere/159/head -> origin/gh/seemethere/159/head 2022-03-17T19:04:58.8062063Z * [new branch] gh/seemethere/159/orig -> origin/gh/seemethere/159/orig 2022-03-17T19:04:58.8063574Z * [new branch] gh/seemethere/16/base -> origin/gh/seemethere/16/base 2022-03-17T19:04:58.8064832Z * [new branch] gh/seemethere/16/orig -> origin/gh/seemethere/16/orig 2022-03-17T19:04:58.8066572Z * [new branch] gh/seemethere/160/base -> origin/gh/seemethere/160/base 2022-03-17T19:04:58.8067835Z * [new branch] gh/seemethere/160/head -> origin/gh/seemethere/160/head 2022-03-17T19:04:58.8069141Z * [new branch] gh/seemethere/160/orig -> origin/gh/seemethere/160/orig 2022-03-17T19:04:58.8070890Z * [new branch] gh/seemethere/188/base -> origin/gh/seemethere/188/base 2022-03-17T19:04:58.8072194Z * [new branch] gh/seemethere/188/head -> origin/gh/seemethere/188/head 2022-03-17T19:04:58.8073541Z * [new branch] gh/seemethere/188/orig -> origin/gh/seemethere/188/orig 2022-03-17T19:04:58.8075403Z * [new branch] gh/seemethere/191/base -> origin/gh/seemethere/191/base 2022-03-17T19:04:58.8076580Z * [new branch] gh/seemethere/191/head -> origin/gh/seemethere/191/head 2022-03-17T19:04:58.8077914Z * [new branch] gh/seemethere/191/orig -> origin/gh/seemethere/191/orig 2022-03-17T19:04:58.8079745Z * [new branch] gh/seemethere/196/base -> origin/gh/seemethere/196/base 2022-03-17T19:04:58.8081054Z * [new branch] gh/seemethere/196/head -> origin/gh/seemethere/196/head 2022-03-17T19:04:58.8082580Z * [new branch] gh/seemethere/196/orig -> origin/gh/seemethere/196/orig 2022-03-17T19:04:58.8084835Z * [new branch] gh/seemethere/197/base -> origin/gh/seemethere/197/base 2022-03-17T19:04:58.8086037Z * [new branch] gh/seemethere/197/head -> origin/gh/seemethere/197/head 2022-03-17T19:04:58.8087339Z * [new branch] gh/seemethere/197/orig -> origin/gh/seemethere/197/orig 2022-03-17T19:04:58.8089124Z * [new branch] gh/seemethere/200/base -> origin/gh/seemethere/200/base 2022-03-17T19:04:58.8090474Z * [new branch] gh/seemethere/200/head -> origin/gh/seemethere/200/head 2022-03-17T19:04:58.8091836Z * [new branch] gh/seemethere/200/orig -> origin/gh/seemethere/200/orig 2022-03-17T19:04:58.8093768Z * [new branch] gh/seemethere/201/base -> origin/gh/seemethere/201/base 2022-03-17T19:04:58.8095015Z * [new branch] gh/seemethere/201/head -> origin/gh/seemethere/201/head 2022-03-17T19:04:58.8096452Z * [new branch] gh/seemethere/201/orig -> origin/gh/seemethere/201/orig 2022-03-17T19:04:58.8098198Z * [new branch] gh/seemethere/207/base -> origin/gh/seemethere/207/base 2022-03-17T19:04:58.8099501Z * [new branch] gh/seemethere/207/head -> origin/gh/seemethere/207/head 2022-03-17T19:04:58.8100967Z * [new branch] gh/seemethere/207/orig -> origin/gh/seemethere/207/orig 2022-03-17T19:04:58.8102627Z * [new branch] gh/seemethere/208/base -> origin/gh/seemethere/208/base 2022-03-17T19:04:58.8103963Z * [new branch] gh/seemethere/208/head -> origin/gh/seemethere/208/head 2022-03-17T19:04:58.8105287Z * [new branch] gh/seemethere/208/orig -> origin/gh/seemethere/208/orig 2022-03-17T19:04:58.8107133Z * [new branch] gh/seemethere/211/base -> origin/gh/seemethere/211/base 2022-03-17T19:04:58.8108519Z * [new branch] gh/seemethere/211/head -> origin/gh/seemethere/211/head 2022-03-17T19:04:58.8109779Z * [new branch] gh/seemethere/211/orig -> origin/gh/seemethere/211/orig 2022-03-17T19:04:58.8111535Z * [new branch] gh/seemethere/215/base -> origin/gh/seemethere/215/base 2022-03-17T19:04:58.8112838Z * [new branch] gh/seemethere/215/orig -> origin/gh/seemethere/215/orig 2022-03-17T19:04:58.8114797Z * [new branch] gh/seemethere/33/base -> origin/gh/seemethere/33/base 2022-03-17T19:04:58.8115992Z * [new branch] gh/seemethere/33/orig -> origin/gh/seemethere/33/orig 2022-03-17T19:04:58.8117651Z * [new branch] gh/seemethere/47/base -> origin/gh/seemethere/47/base 2022-03-17T19:04:58.8118953Z * [new branch] gh/seemethere/47/orig -> origin/gh/seemethere/47/orig 2022-03-17T19:04:58.8120733Z * [new branch] gh/seemethere/8/base -> origin/gh/seemethere/8/base 2022-03-17T19:04:58.8122113Z * [new branch] gh/seemethere/8/orig -> origin/gh/seemethere/8/orig 2022-03-17T19:04:58.8123915Z * [new branch] gh/seemethere/83/base -> origin/gh/seemethere/83/base 2022-03-17T19:04:58.8125196Z * [new branch] gh/seemethere/83/head -> origin/gh/seemethere/83/head 2022-03-17T19:04:58.8126538Z * [new branch] gh/seemethere/83/orig -> origin/gh/seemethere/83/orig 2022-03-17T19:04:58.8128393Z * [new branch] gh/seemethere/90/base -> origin/gh/seemethere/90/base 2022-03-17T19:04:58.8129715Z * [new branch] gh/seemethere/90/head -> origin/gh/seemethere/90/head 2022-03-17T19:04:58.8130982Z * [new branch] gh/seemethere/90/orig -> origin/gh/seemethere/90/orig 2022-03-17T19:04:58.8133024Z * [new branch] gh/shunting314/13/base -> origin/gh/shunting314/13/base 2022-03-17T19:04:58.8134288Z * [new branch] gh/shunting314/13/head -> origin/gh/shunting314/13/head 2022-03-17T19:04:58.8135855Z * [new branch] gh/shunting314/14/base -> origin/gh/shunting314/14/base 2022-03-17T19:04:58.8137341Z * [new branch] gh/shunting314/14/head -> origin/gh/shunting314/14/head 2022-03-17T19:04:58.8138693Z * [new branch] gh/shunting314/14/orig -> origin/gh/shunting314/14/orig 2022-03-17T19:04:58.8140403Z * [new branch] gh/shunting314/16/base -> origin/gh/shunting314/16/base 2022-03-17T19:04:58.8141778Z * [new branch] gh/shunting314/16/head -> origin/gh/shunting314/16/head 2022-03-17T19:04:58.8143078Z * [new branch] gh/shunting314/16/orig -> origin/gh/shunting314/16/orig 2022-03-17T19:04:58.8144987Z * [new branch] gh/shunting314/17/base -> origin/gh/shunting314/17/base 2022-03-17T19:04:58.8146236Z * [new branch] gh/shunting314/17/head -> origin/gh/shunting314/17/head 2022-03-17T19:04:58.8147453Z * [new branch] gh/shunting314/17/orig -> origin/gh/shunting314/17/orig 2022-03-17T19:04:58.8149177Z * [new branch] gh/shunting314/18/base -> origin/gh/shunting314/18/base 2022-03-17T19:04:58.8150642Z * [new branch] gh/shunting314/18/head -> origin/gh/shunting314/18/head 2022-03-17T19:04:58.8151945Z * [new branch] gh/shunting314/18/orig -> origin/gh/shunting314/18/orig 2022-03-17T19:04:58.8153707Z * [new branch] gh/shunting314/6/base -> origin/gh/shunting314/6/base 2022-03-17T19:04:58.8155063Z * [new branch] gh/shunting314/6/head -> origin/gh/shunting314/6/head 2022-03-17T19:04:58.8156825Z * [new branch] gh/shunting314/9/base -> origin/gh/shunting314/9/base 2022-03-17T19:04:58.8158156Z * [new branch] gh/shunting314/9/head -> origin/gh/shunting314/9/head 2022-03-17T19:04:58.8159487Z * [new branch] gh/shunting314/9/orig -> origin/gh/shunting314/9/orig 2022-03-17T19:04:58.8161703Z * [new branch] gh/simpkins/1/base -> origin/gh/simpkins/1/base 2022-03-17T19:04:58.8164378Z * [new branch] gh/simpkins/1/orig -> origin/gh/simpkins/1/orig 2022-03-17T19:04:58.8166008Z * [new branch] gh/simpkins/2/base -> origin/gh/simpkins/2/base 2022-03-17T19:04:58.8167400Z * [new branch] gh/simpkins/2/head -> origin/gh/simpkins/2/head 2022-03-17T19:04:58.8168585Z * [new branch] gh/simpkins/2/orig -> origin/gh/simpkins/2/orig 2022-03-17T19:04:58.8170672Z * [new branch] gh/sinannasir/12/base -> origin/gh/sinannasir/12/base 2022-03-17T19:04:58.8172736Z * [new branch] gh/sinannasir/12/orig -> origin/gh/sinannasir/12/orig 2022-03-17T19:04:58.8174492Z * [new branch] gh/sinannasir/8/base -> origin/gh/sinannasir/8/base 2022-03-17T19:04:58.8176200Z * [new branch] gh/sinannasir/9/base -> origin/gh/sinannasir/9/base 2022-03-17T19:04:58.8177655Z * [new branch] gh/sinannasir/9/head -> origin/gh/sinannasir/9/head 2022-03-17T19:04:58.8179195Z * [new branch] gh/sinannasir/9/orig -> origin/gh/sinannasir/9/orig 2022-03-17T19:04:58.8181120Z * [new branch] gh/smessmer/124/base -> origin/gh/smessmer/124/base 2022-03-17T19:04:58.8182473Z * [new branch] gh/smessmer/124/orig -> origin/gh/smessmer/124/orig 2022-03-17T19:04:58.8184250Z * [new branch] gh/smessmer/131/base -> origin/gh/smessmer/131/base 2022-03-17T19:04:58.8185617Z * [new branch] gh/smessmer/131/head -> origin/gh/smessmer/131/head 2022-03-17T19:04:58.8187093Z * [new branch] gh/smessmer/131/orig -> origin/gh/smessmer/131/orig 2022-03-17T19:04:58.8188856Z * [new branch] gh/smessmer/139/base -> origin/gh/smessmer/139/base 2022-03-17T19:04:58.8190260Z * [new branch] gh/smessmer/139/head -> origin/gh/smessmer/139/head 2022-03-17T19:04:58.8191625Z * [new branch] gh/smessmer/139/orig -> origin/gh/smessmer/139/orig 2022-03-17T19:04:58.8193260Z * [new branch] gh/smessmer/141/base -> origin/gh/smessmer/141/base 2022-03-17T19:04:58.8194643Z * [new branch] gh/smessmer/141/head -> origin/gh/smessmer/141/head 2022-03-17T19:04:58.8195984Z * [new branch] gh/smessmer/141/orig -> origin/gh/smessmer/141/orig 2022-03-17T19:04:58.8197750Z * [new branch] gh/smessmer/15/base -> origin/gh/smessmer/15/base 2022-03-17T19:04:58.8199308Z * [new branch] gh/smessmer/15/head -> origin/gh/smessmer/15/head 2022-03-17T19:04:58.8200768Z * [new branch] gh/smessmer/15/orig -> origin/gh/smessmer/15/orig 2022-03-17T19:04:58.8202662Z * [new branch] gh/smessmer/157/base -> origin/gh/smessmer/157/base 2022-03-17T19:04:58.8204209Z * [new branch] gh/smessmer/157/head -> origin/gh/smessmer/157/head 2022-03-17T19:04:58.8205827Z * [new branch] gh/smessmer/157/orig -> origin/gh/smessmer/157/orig 2022-03-17T19:04:58.8207315Z * [new branch] gh/smessmer/158/base -> origin/gh/smessmer/158/base 2022-03-17T19:04:58.8208450Z * [new branch] gh/smessmer/158/head -> origin/gh/smessmer/158/head 2022-03-17T19:04:58.8209953Z * [new branch] gh/smessmer/158/orig -> origin/gh/smessmer/158/orig 2022-03-17T19:04:58.8211624Z * [new branch] gh/smessmer/159/base -> origin/gh/smessmer/159/base 2022-03-17T19:04:58.8212812Z * [new branch] gh/smessmer/159/head -> origin/gh/smessmer/159/head 2022-03-17T19:04:58.8214245Z * [new branch] gh/smessmer/159/orig -> origin/gh/smessmer/159/orig 2022-03-17T19:04:58.8216045Z * [new branch] gh/smessmer/160/base -> origin/gh/smessmer/160/base 2022-03-17T19:04:58.8216953Z * [new branch] gh/smessmer/160/head -> origin/gh/smessmer/160/head 2022-03-17T19:04:58.8218507Z * [new branch] gh/smessmer/160/orig -> origin/gh/smessmer/160/orig 2022-03-17T19:04:58.8220027Z * [new branch] gh/smessmer/161/base -> origin/gh/smessmer/161/base 2022-03-17T19:04:58.8221124Z * [new branch] gh/smessmer/161/head -> origin/gh/smessmer/161/head 2022-03-17T19:04:58.8222602Z * [new branch] gh/smessmer/161/orig -> origin/gh/smessmer/161/orig 2022-03-17T19:04:58.8224449Z * [new branch] gh/smessmer/162/orig -> origin/gh/smessmer/162/orig 2022-03-17T19:04:58.8226037Z * [new branch] gh/smessmer/164/base -> origin/gh/smessmer/164/base 2022-03-17T19:04:58.8227149Z * [new branch] gh/smessmer/164/head -> origin/gh/smessmer/164/head 2022-03-17T19:04:58.8228768Z * [new branch] gh/smessmer/164/orig -> origin/gh/smessmer/164/orig 2022-03-17T19:04:58.8230401Z * [new branch] gh/smessmer/175/base -> origin/gh/smessmer/175/base 2022-03-17T19:04:58.8231858Z * [new branch] gh/smessmer/175/orig -> origin/gh/smessmer/175/orig 2022-03-17T19:04:58.8233557Z * [new branch] gh/smessmer/181/base -> origin/gh/smessmer/181/base 2022-03-17T19:04:58.8235009Z * [new branch] gh/smessmer/181/head -> origin/gh/smessmer/181/head 2022-03-17T19:04:58.8236118Z * [new branch] gh/smessmer/181/orig -> origin/gh/smessmer/181/orig 2022-03-17T19:04:58.8238001Z * [new branch] gh/smessmer/21/base -> origin/gh/smessmer/21/base 2022-03-17T19:04:58.8239348Z * [new branch] gh/smessmer/21/head -> origin/gh/smessmer/21/head 2022-03-17T19:04:58.8240808Z * [new branch] gh/smessmer/21/orig -> origin/gh/smessmer/21/orig 2022-03-17T19:04:58.8242565Z * [new branch] gh/smessmer/210/base -> origin/gh/smessmer/210/base 2022-03-17T19:04:58.8244117Z * [new branch] gh/smessmer/210/head -> origin/gh/smessmer/210/head 2022-03-17T19:04:58.8245627Z * [new branch] gh/smessmer/210/orig -> origin/gh/smessmer/210/orig 2022-03-17T19:04:58.8247214Z * [new branch] gh/smessmer/227/base -> origin/gh/smessmer/227/base 2022-03-17T19:04:58.8248657Z * [new branch] gh/smessmer/227/head -> origin/gh/smessmer/227/head 2022-03-17T19:04:58.8250361Z * [new branch] gh/smessmer/242/base -> origin/gh/smessmer/242/base 2022-03-17T19:04:58.8251755Z * [new branch] gh/smessmer/242/head -> origin/gh/smessmer/242/head 2022-03-17T19:04:58.8252784Z * [new branch] gh/smessmer/242/orig -> origin/gh/smessmer/242/orig 2022-03-17T19:04:58.8254709Z * [new branch] gh/smessmer/243/base -> origin/gh/smessmer/243/base 2022-03-17T19:04:58.8256158Z * [new branch] gh/smessmer/243/head -> origin/gh/smessmer/243/head 2022-03-17T19:04:58.8257329Z * [new branch] gh/smessmer/243/orig -> origin/gh/smessmer/243/orig 2022-03-17T19:04:58.8258984Z * [new branch] gh/smessmer/244/base -> origin/gh/smessmer/244/base 2022-03-17T19:04:58.8260421Z * [new branch] gh/smessmer/244/head -> origin/gh/smessmer/244/head 2022-03-17T19:04:58.8261572Z * [new branch] gh/smessmer/244/orig -> origin/gh/smessmer/244/orig 2022-03-17T19:04:58.8263563Z * [new branch] gh/smessmer/263/base -> origin/gh/smessmer/263/base 2022-03-17T19:04:58.8265079Z * [new branch] gh/smessmer/263/head -> origin/gh/smessmer/263/head 2022-03-17T19:04:58.8266424Z * [new branch] gh/smessmer/263/orig -> origin/gh/smessmer/263/orig 2022-03-17T19:04:58.8268603Z * [new branch] gh/smessmer/282/base -> origin/gh/smessmer/282/base 2022-03-17T19:04:58.8270051Z * [new branch] gh/smessmer/282/orig -> origin/gh/smessmer/282/orig 2022-03-17T19:04:58.8271748Z * [new branch] gh/smessmer/294/base -> origin/gh/smessmer/294/base 2022-03-17T19:04:58.8273144Z * [new branch] gh/smessmer/294/orig -> origin/gh/smessmer/294/orig 2022-03-17T19:04:58.8274928Z * [new branch] gh/smessmer/296/base -> origin/gh/smessmer/296/base 2022-03-17T19:04:58.8276092Z * [new branch] gh/smessmer/296/head -> origin/gh/smessmer/296/head 2022-03-17T19:04:58.8277767Z * [new branch] gh/smessmer/296/orig -> origin/gh/smessmer/296/orig 2022-03-17T19:04:58.8279478Z * [new branch] gh/smessmer/297/base -> origin/gh/smessmer/297/base 2022-03-17T19:04:58.8280864Z * [new branch] gh/smessmer/297/head -> origin/gh/smessmer/297/head 2022-03-17T19:04:58.8282193Z * [new branch] gh/smessmer/297/orig -> origin/gh/smessmer/297/orig 2022-03-17T19:04:58.8284157Z * [new branch] gh/smessmer/298/base -> origin/gh/smessmer/298/base 2022-03-17T19:04:58.8285206Z * [new branch] gh/smessmer/298/head -> origin/gh/smessmer/298/head 2022-03-17T19:04:58.8286673Z * [new branch] gh/smessmer/298/orig -> origin/gh/smessmer/298/orig 2022-03-17T19:04:58.8288513Z * [new branch] gh/smessmer/30/base -> origin/gh/smessmer/30/base 2022-03-17T19:04:58.8289899Z * [new branch] gh/smessmer/30/orig -> origin/gh/smessmer/30/orig 2022-03-17T19:04:58.8291455Z * [new branch] gh/smessmer/307/base -> origin/gh/smessmer/307/base 2022-03-17T19:04:58.8292767Z * [new branch] gh/smessmer/307/head -> origin/gh/smessmer/307/head 2022-03-17T19:04:58.8294596Z * [new branch] gh/smessmer/307/orig -> origin/gh/smessmer/307/orig 2022-03-17T19:04:58.8295980Z * [new branch] gh/smessmer/310/base -> origin/gh/smessmer/310/base 2022-03-17T19:04:58.8297449Z * [new branch] gh/smessmer/310/head -> origin/gh/smessmer/310/head 2022-03-17T19:04:58.8298755Z * [new branch] gh/smessmer/310/orig -> origin/gh/smessmer/310/orig 2022-03-17T19:04:58.8300477Z * [new branch] gh/smessmer/34/base -> origin/gh/smessmer/34/base 2022-03-17T19:04:58.8301865Z * [new branch] gh/smessmer/34/orig -> origin/gh/smessmer/34/orig 2022-03-17T19:04:58.8303803Z * [new branch] gh/smessmer/36/orig -> origin/gh/smessmer/36/orig 2022-03-17T19:04:58.8305593Z * [new branch] gh/smessmer/39/base -> origin/gh/smessmer/39/base 2022-03-17T19:04:58.8306737Z * [new branch] gh/smessmer/39/orig -> origin/gh/smessmer/39/orig 2022-03-17T19:04:58.8308540Z * [new branch] gh/smessmer/56/base -> origin/gh/smessmer/56/base 2022-03-17T19:04:58.8309769Z * [new branch] gh/smessmer/56/orig -> origin/gh/smessmer/56/orig 2022-03-17T19:04:58.8311572Z * [new branch] gh/smessmer/61/base -> origin/gh/smessmer/61/base 2022-03-17T19:04:58.8312690Z * [new branch] gh/smessmer/61/head -> origin/gh/smessmer/61/head 2022-03-17T19:04:58.8313971Z * [new branch] gh/smessmer/61/orig -> origin/gh/smessmer/61/orig 2022-03-17T19:04:58.8315786Z * [new branch] gh/smessmer/68/base -> origin/gh/smessmer/68/base 2022-03-17T19:04:58.8317147Z * [new branch] gh/smessmer/68/head -> origin/gh/smessmer/68/head 2022-03-17T19:04:58.8318521Z * [new branch] gh/smessmer/68/orig -> origin/gh/smessmer/68/orig 2022-03-17T19:04:58.8320287Z * [new branch] gh/smessmer/7/base -> origin/gh/smessmer/7/base 2022-03-17T19:04:58.8322014Z * [new branch] gh/smessmer/7/head -> origin/gh/smessmer/7/head 2022-03-17T19:04:58.8324612Z * [new branch] gh/smessmer/72/head -> origin/gh/smessmer/72/head 2022-03-17T19:04:58.8325865Z * [new branch] gh/smessmer/72/orig -> origin/gh/smessmer/72/orig 2022-03-17T19:04:58.8327418Z * [new branch] gh/smessmer/8/base -> origin/gh/smessmer/8/base 2022-03-17T19:04:58.8329130Z * [new branch] gh/smessmer/81/base -> origin/gh/smessmer/81/base 2022-03-17T19:04:58.8330446Z * [new branch] gh/smessmer/81/orig -> origin/gh/smessmer/81/orig 2022-03-17T19:04:58.8332271Z * [new branch] gh/smessmer/92/base -> origin/gh/smessmer/92/base 2022-03-17T19:04:58.8333804Z * [new branch] gh/smessmer/92/orig -> origin/gh/smessmer/92/orig 2022-03-17T19:04:58.8335755Z * [new branch] gh/soulitzer/23/base -> origin/gh/soulitzer/23/base 2022-03-17T19:04:58.8337067Z * [new branch] gh/soulitzer/23/head -> origin/gh/soulitzer/23/head 2022-03-17T19:04:58.8338305Z * [new branch] gh/soulitzer/23/orig -> origin/gh/soulitzer/23/orig 2022-03-17T19:04:58.8340077Z * [new branch] gh/soulitzer/24/base -> origin/gh/soulitzer/24/base 2022-03-17T19:04:58.8341530Z * [new branch] gh/soulitzer/24/head -> origin/gh/soulitzer/24/head 2022-03-17T19:04:58.8342819Z * [new branch] gh/soulitzer/24/orig -> origin/gh/soulitzer/24/orig 2022-03-17T19:04:58.8344469Z * [new branch] gh/soulitzer/25/base -> origin/gh/soulitzer/25/base 2022-03-17T19:04:58.8345741Z * [new branch] gh/soulitzer/25/head -> origin/gh/soulitzer/25/head 2022-03-17T19:04:58.8347068Z * [new branch] gh/soulitzer/25/orig -> origin/gh/soulitzer/25/orig 2022-03-17T19:04:58.8348676Z * [new branch] gh/soulitzer/27/base -> origin/gh/soulitzer/27/base 2022-03-17T19:04:58.8349981Z * [new branch] gh/soulitzer/27/head -> origin/gh/soulitzer/27/head 2022-03-17T19:04:58.8351203Z * [new branch] gh/soulitzer/27/orig -> origin/gh/soulitzer/27/orig 2022-03-17T19:04:58.8353292Z * [new branch] gh/soulitzer/49/base -> origin/gh/soulitzer/49/base 2022-03-17T19:04:58.8354514Z * [new branch] gh/soulitzer/49/head -> origin/gh/soulitzer/49/head 2022-03-17T19:04:58.8355870Z * [new branch] gh/soulitzer/49/orig -> origin/gh/soulitzer/49/orig 2022-03-17T19:04:58.8357565Z * [new branch] gh/soulitzer/57/base -> origin/gh/soulitzer/57/base 2022-03-17T19:04:58.8358906Z * [new branch] gh/soulitzer/57/head -> origin/gh/soulitzer/57/head 2022-03-17T19:04:58.8360242Z * [new branch] gh/soulitzer/57/orig -> origin/gh/soulitzer/57/orig 2022-03-17T19:04:58.8361537Z * [new branch] gh/soulitzer/58/base -> origin/gh/soulitzer/58/base 2022-03-17T19:04:58.8363379Z * [new branch] gh/soulitzer/58/head -> origin/gh/soulitzer/58/head 2022-03-17T19:04:58.8364603Z * [new branch] gh/soulitzer/58/orig -> origin/gh/soulitzer/58/orig 2022-03-17T19:04:58.8366342Z * [new branch] gh/soulitzer/59/base -> origin/gh/soulitzer/59/base 2022-03-17T19:04:58.8367640Z * [new branch] gh/soulitzer/59/head -> origin/gh/soulitzer/59/head 2022-03-17T19:04:58.8368835Z * [new branch] gh/soulitzer/59/orig -> origin/gh/soulitzer/59/orig 2022-03-17T19:04:58.8370604Z * [new branch] gh/soulitzer/64/base -> origin/gh/soulitzer/64/base 2022-03-17T19:04:58.8371785Z * [new branch] gh/soulitzer/64/head -> origin/gh/soulitzer/64/head 2022-03-17T19:04:58.8373131Z * [new branch] gh/soulitzer/64/orig -> origin/gh/soulitzer/64/orig 2022-03-17T19:04:58.8374987Z * [new branch] gh/soulitzer/65/base -> origin/gh/soulitzer/65/base 2022-03-17T19:04:58.8376573Z * [new branch] gh/soulitzer/65/head -> origin/gh/soulitzer/65/head 2022-03-17T19:04:58.8377807Z * [new branch] gh/soulitzer/65/orig -> origin/gh/soulitzer/65/orig 2022-03-17T19:04:58.8379589Z * [new branch] gh/soulitzer/66/base -> origin/gh/soulitzer/66/base 2022-03-17T19:04:58.8380944Z * [new branch] gh/soulitzer/66/head -> origin/gh/soulitzer/66/head 2022-03-17T19:04:58.8382274Z * [new branch] gh/soulitzer/66/orig -> origin/gh/soulitzer/66/orig 2022-03-17T19:04:58.8384041Z * [new branch] gh/soulitzer/67/base -> origin/gh/soulitzer/67/base 2022-03-17T19:04:58.8385383Z * [new branch] gh/soulitzer/67/head -> origin/gh/soulitzer/67/head 2022-03-17T19:04:58.8386930Z * [new branch] gh/soulitzer/67/orig -> origin/gh/soulitzer/67/orig 2022-03-17T19:04:58.8388824Z * [new branch] gh/splitinfinity/105/base -> origin/gh/splitinfinity/105/base 2022-03-17T19:04:58.8390024Z * [new branch] gh/splitinfinity/105/head -> origin/gh/splitinfinity/105/head 2022-03-17T19:04:58.8391322Z * [new branch] gh/splitinfinity/105/orig -> origin/gh/splitinfinity/105/orig 2022-03-17T19:04:58.8393219Z * [new branch] gh/splitinfinity/114/base -> origin/gh/splitinfinity/114/base 2022-03-17T19:04:58.8394628Z * [new branch] gh/splitinfinity/114/head -> origin/gh/splitinfinity/114/head 2022-03-17T19:04:58.8395997Z * [new branch] gh/splitinfinity/114/orig -> origin/gh/splitinfinity/114/orig 2022-03-17T19:04:58.8397845Z * [new branch] gh/splitinfinity/116/base -> origin/gh/splitinfinity/116/base 2022-03-17T19:04:58.8399108Z * [new branch] gh/splitinfinity/116/head -> origin/gh/splitinfinity/116/head 2022-03-17T19:04:58.8400476Z * [new branch] gh/splitinfinity/116/orig -> origin/gh/splitinfinity/116/orig 2022-03-17T19:04:58.8402340Z * [new branch] gh/splitinfinity/12/base -> origin/gh/splitinfinity/12/base 2022-03-17T19:04:58.8404294Z * [new branch] gh/splitinfinity/12/head -> origin/gh/splitinfinity/12/head 2022-03-17T19:04:58.8405581Z * [new branch] gh/splitinfinity/12/orig -> origin/gh/splitinfinity/12/orig 2022-03-17T19:04:58.8407483Z * [new branch] gh/splitinfinity/125/base -> origin/gh/splitinfinity/125/base 2022-03-17T19:04:58.8408793Z * [new branch] gh/splitinfinity/125/head -> origin/gh/splitinfinity/125/head 2022-03-17T19:04:58.8410044Z * [new branch] gh/splitinfinity/125/orig -> origin/gh/splitinfinity/125/orig 2022-03-17T19:04:58.8411937Z * [new branch] gh/splitinfinity/126/base -> origin/gh/splitinfinity/126/base 2022-03-17T19:04:58.8413174Z * [new branch] gh/splitinfinity/126/head -> origin/gh/splitinfinity/126/head 2022-03-17T19:04:58.8414456Z * [new branch] gh/splitinfinity/126/orig -> origin/gh/splitinfinity/126/orig 2022-03-17T19:04:58.8416261Z * [new branch] gh/splitinfinity/130/base -> origin/gh/splitinfinity/130/base 2022-03-17T19:04:58.8417578Z * [new branch] gh/splitinfinity/130/head -> origin/gh/splitinfinity/130/head 2022-03-17T19:04:58.8418928Z * [new branch] gh/splitinfinity/130/orig -> origin/gh/splitinfinity/130/orig 2022-03-17T19:04:58.8420780Z * [new branch] gh/splitinfinity/136/base -> origin/gh/splitinfinity/136/base 2022-03-17T19:04:58.8422109Z * [new branch] gh/splitinfinity/136/head -> origin/gh/splitinfinity/136/head 2022-03-17T19:04:58.8423472Z * [new branch] gh/splitinfinity/136/orig -> origin/gh/splitinfinity/136/orig 2022-03-17T19:04:58.8425298Z * [new branch] gh/splitinfinity/137/base -> origin/gh/splitinfinity/137/base 2022-03-17T19:04:58.8426663Z * [new branch] gh/splitinfinity/137/head -> origin/gh/splitinfinity/137/head 2022-03-17T19:04:58.8427984Z * [new branch] gh/splitinfinity/137/orig -> origin/gh/splitinfinity/137/orig 2022-03-17T19:04:58.8429323Z * [new branch] gh/splitinfinity/144/base -> origin/gh/splitinfinity/144/base 2022-03-17T19:04:58.8430674Z * [new branch] gh/splitinfinity/144/head -> origin/gh/splitinfinity/144/head 2022-03-17T19:04:58.8432022Z * [new branch] gh/splitinfinity/144/orig -> origin/gh/splitinfinity/144/orig 2022-03-17T19:04:58.8433757Z * [new branch] gh/splitinfinity/155/base -> origin/gh/splitinfinity/155/base 2022-03-17T19:04:58.8435700Z * [new branch] gh/splitinfinity/155/head -> origin/gh/splitinfinity/155/head 2022-03-17T19:04:58.8436694Z * [new branch] gh/splitinfinity/155/orig -> origin/gh/splitinfinity/155/orig 2022-03-17T19:04:58.8438581Z * [new branch] gh/splitinfinity/156/base -> origin/gh/splitinfinity/156/base 2022-03-17T19:04:58.8439638Z * [new branch] gh/splitinfinity/156/head -> origin/gh/splitinfinity/156/head 2022-03-17T19:04:58.8441054Z * [new branch] gh/splitinfinity/156/orig -> origin/gh/splitinfinity/156/orig 2022-03-17T19:04:58.8443617Z * [new branch] gh/splitinfinity/163/base -> origin/gh/splitinfinity/163/base 2022-03-17T19:04:58.8444333Z * [new branch] gh/splitinfinity/163/head -> origin/gh/splitinfinity/163/head 2022-03-17T19:04:58.8446045Z * [new branch] gh/splitinfinity/163/orig -> origin/gh/splitinfinity/163/orig 2022-03-17T19:04:58.8447218Z * [new branch] gh/splitinfinity/23/base -> origin/gh/splitinfinity/23/base 2022-03-17T19:04:58.8448758Z * [new branch] gh/splitinfinity/23/orig -> origin/gh/splitinfinity/23/orig 2022-03-17T19:04:58.8450489Z * [new branch] gh/splitinfinity/24/base -> origin/gh/splitinfinity/24/base 2022-03-17T19:04:58.8451865Z * [new branch] gh/splitinfinity/24/orig -> origin/gh/splitinfinity/24/orig 2022-03-17T19:04:58.8453597Z * [new branch] gh/splitinfinity/33/base -> origin/gh/splitinfinity/33/base 2022-03-17T19:04:58.8454782Z * [new branch] gh/splitinfinity/33/head -> origin/gh/splitinfinity/33/head 2022-03-17T19:04:58.8456450Z * [new branch] gh/splitinfinity/33/orig -> origin/gh/splitinfinity/33/orig 2022-03-17T19:04:58.8457982Z * [new branch] gh/splitinfinity/60/base -> origin/gh/splitinfinity/60/base 2022-03-17T19:04:58.8459584Z * [new branch] gh/splitinfinity/61/base -> origin/gh/splitinfinity/61/base 2022-03-17T19:04:58.8461154Z * [new branch] gh/splitinfinity/67/base -> origin/gh/splitinfinity/67/base 2022-03-17T19:04:58.8462326Z * [new branch] gh/splitinfinity/67/head -> origin/gh/splitinfinity/67/head 2022-03-17T19:04:58.8463983Z * [new branch] gh/splitinfinity/67/orig -> origin/gh/splitinfinity/67/orig 2022-03-17T19:04:58.8465666Z * [new branch] gh/splitinfinity/7/base -> origin/gh/splitinfinity/7/base 2022-03-17T19:04:58.8467159Z * [new branch] gh/splitinfinity/7/head -> origin/gh/splitinfinity/7/head 2022-03-17T19:04:58.8468314Z * [new branch] gh/splitinfinity/7/orig -> origin/gh/splitinfinity/7/orig 2022-03-17T19:04:58.8470143Z * [new branch] gh/splitinfinity/8/base -> origin/gh/splitinfinity/8/base 2022-03-17T19:04:58.8471586Z * [new branch] gh/splitinfinity/8/head -> origin/gh/splitinfinity/8/head 2022-03-17T19:04:58.8473073Z * [new branch] gh/splitinfinity/8/orig -> origin/gh/splitinfinity/8/orig 2022-03-17T19:04:58.8474749Z * [new branch] gh/splitinfinity/81/base -> origin/gh/splitinfinity/81/base 2022-03-17T19:04:58.8476226Z * [new branch] gh/splitinfinity/81/head -> origin/gh/splitinfinity/81/head 2022-03-17T19:04:58.8477440Z * [new branch] gh/splitinfinity/81/orig -> origin/gh/splitinfinity/81/orig 2022-03-17T19:04:58.8479327Z * [new branch] gh/splitinfinity/86/base -> origin/gh/splitinfinity/86/base 2022-03-17T19:04:58.8480821Z * [new branch] gh/splitinfinity/86/head -> origin/gh/splitinfinity/86/head 2022-03-17T19:04:58.8482398Z * [new branch] gh/splitinfinity/86/orig -> origin/gh/splitinfinity/86/orig 2022-03-17T19:04:58.8484814Z * [new branch] gh/splitinfinity/93/base -> origin/gh/splitinfinity/93/base 2022-03-17T19:04:58.8485936Z * [new branch] gh/splitinfinity/93/head -> origin/gh/splitinfinity/93/head 2022-03-17T19:04:58.8487387Z * [new branch] gh/splitinfinity/93/orig -> origin/gh/splitinfinity/93/orig 2022-03-17T19:04:58.8489282Z * [new branch] gh/splitinfinity/99/base -> origin/gh/splitinfinity/99/base 2022-03-17T19:04:58.8490729Z * [new branch] gh/splitinfinity/99/head -> origin/gh/splitinfinity/99/head 2022-03-17T19:04:58.8492233Z * [new branch] gh/splitinfinity/99/orig -> origin/gh/splitinfinity/99/orig 2022-03-17T19:04:58.8494139Z * [new branch] gh/stngo98/1/base -> origin/gh/stngo98/1/base 2022-03-17T19:04:58.8495252Z * [new branch] gh/stngo98/1/head -> origin/gh/stngo98/1/head 2022-03-17T19:04:58.8496754Z * [new branch] gh/stngo98/1/orig -> origin/gh/stngo98/1/orig 2022-03-17T19:04:58.8498461Z * [new branch] gh/stngo98/2/base -> origin/gh/stngo98/2/base 2022-03-17T19:04:58.8499623Z * [new branch] gh/stngo98/2/head -> origin/gh/stngo98/2/head 2022-03-17T19:04:58.8501136Z * [new branch] gh/stngo98/2/orig -> origin/gh/stngo98/2/orig 2022-03-17T19:04:58.8502974Z * [new branch] gh/suo/104/base -> origin/gh/suo/104/base 2022-03-17T19:04:58.8504721Z * [new branch] gh/suo/121/base -> origin/gh/suo/121/base 2022-03-17T19:04:58.8505954Z * [new branch] gh/suo/121/orig -> origin/gh/suo/121/orig 2022-03-17T19:04:58.8507672Z * [new branch] gh/suo/132/head -> origin/gh/suo/132/head 2022-03-17T19:04:58.8509412Z * [new branch] gh/suo/149/base -> origin/gh/suo/149/base 2022-03-17T19:04:58.8510760Z * [new branch] gh/suo/149/orig -> origin/gh/suo/149/orig 2022-03-17T19:04:58.8512470Z * [new branch] gh/suo/152/base -> origin/gh/suo/152/base 2022-03-17T19:04:58.8513876Z * [new branch] gh/suo/152/head -> origin/gh/suo/152/head 2022-03-17T19:04:58.8515286Z * [new branch] gh/suo/152/orig -> origin/gh/suo/152/orig 2022-03-17T19:04:58.8516910Z * [new branch] gh/suo/157/base -> origin/gh/suo/157/base 2022-03-17T19:04:58.8518036Z * [new branch] gh/suo/157/head -> origin/gh/suo/157/head 2022-03-17T19:04:58.8519925Z * [new branch] gh/suo/159/base -> origin/gh/suo/159/base 2022-03-17T19:04:58.8521092Z * [new branch] gh/suo/159/orig -> origin/gh/suo/159/orig 2022-03-17T19:04:58.8523342Z * [new branch] gh/suo/166/base -> origin/gh/suo/166/base 2022-03-17T19:04:58.8524467Z * [new branch] gh/suo/166/orig -> origin/gh/suo/166/orig 2022-03-17T19:04:58.8526514Z * [new branch] gh/suo/169/base -> origin/gh/suo/169/base 2022-03-17T19:04:58.8527964Z * [new branch] gh/suo/169/orig -> origin/gh/suo/169/orig 2022-03-17T19:04:58.8529704Z * [new branch] gh/suo/172/base -> origin/gh/suo/172/base 2022-03-17T19:04:58.8530734Z * [new branch] gh/suo/172/orig -> origin/gh/suo/172/orig 2022-03-17T19:04:58.8532372Z * [new branch] gh/suo/181/base -> origin/gh/suo/181/base 2022-03-17T19:04:58.8534807Z * [new branch] gh/suo/181/orig -> origin/gh/suo/181/orig 2022-03-17T19:04:58.8535826Z * [new branch] gh/suo/193/base -> origin/gh/suo/193/base 2022-03-17T19:04:58.8536952Z * [new branch] gh/suo/193/head -> origin/gh/suo/193/head 2022-03-17T19:04:58.8538333Z * [new branch] gh/suo/193/orig -> origin/gh/suo/193/orig 2022-03-17T19:04:58.8540345Z * [new branch] gh/suo/201/base -> origin/gh/suo/201/base 2022-03-17T19:04:58.8541364Z * [new branch] gh/suo/201/head -> origin/gh/suo/201/head 2022-03-17T19:04:58.8543014Z * [new branch] gh/suo/210/base -> origin/gh/suo/210/base 2022-03-17T19:04:58.8544459Z * [new branch] gh/suo/210/head -> origin/gh/suo/210/head 2022-03-17T19:04:58.8545577Z * [new branch] gh/suo/210/orig -> origin/gh/suo/210/orig 2022-03-17T19:04:58.8547492Z * [new branch] gh/suo/220/head -> origin/gh/suo/220/head 2022-03-17T19:04:58.8549142Z * [new branch] gh/suo/220/orig -> origin/gh/suo/220/orig 2022-03-17T19:04:58.8550756Z * [new branch] gh/suo/227/base -> origin/gh/suo/227/base 2022-03-17T19:04:58.8551900Z * [new branch] gh/suo/227/head -> origin/gh/suo/227/head 2022-03-17T19:04:58.8553399Z * [new branch] gh/suo/227/orig -> origin/gh/suo/227/orig 2022-03-17T19:04:58.8555134Z * [new branch] gh/suo/234/base -> origin/gh/suo/234/base 2022-03-17T19:04:58.8556794Z * [new branch] gh/suo/260/base -> origin/gh/suo/260/base 2022-03-17T19:04:58.8557923Z * [new branch] gh/suo/260/head -> origin/gh/suo/260/head 2022-03-17T19:04:58.8559351Z * [new branch] gh/suo/260/orig -> origin/gh/suo/260/orig 2022-03-17T19:04:58.8561037Z * [new branch] gh/suo/291/base -> origin/gh/suo/291/base 2022-03-17T19:04:58.8562553Z * [new branch] gh/suo/291/orig -> origin/gh/suo/291/orig 2022-03-17T19:04:58.8564413Z * [new branch] gh/suo/301/base -> origin/gh/suo/301/base 2022-03-17T19:04:58.8565709Z * [new branch] gh/suo/301/head -> origin/gh/suo/301/head 2022-03-17T19:04:58.8567046Z * [new branch] gh/suo/301/orig -> origin/gh/suo/301/orig 2022-03-17T19:04:58.8568734Z * [new branch] gh/suo/332/base -> origin/gh/suo/332/base 2022-03-17T19:04:58.8570124Z * [new branch] gh/suo/332/head -> origin/gh/suo/332/head 2022-03-17T19:04:58.8571533Z * [new branch] gh/suo/332/orig -> origin/gh/suo/332/orig 2022-03-17T19:04:58.8573234Z * [new branch] gh/suo/342/base -> origin/gh/suo/342/base 2022-03-17T19:04:58.8574587Z * [new branch] gh/suo/342/head -> origin/gh/suo/342/head 2022-03-17T19:04:58.8576016Z * [new branch] gh/suo/342/orig -> origin/gh/suo/342/orig 2022-03-17T19:04:58.8577772Z * [new branch] gh/suo/351/base -> origin/gh/suo/351/base 2022-03-17T19:04:58.8579146Z * [new branch] gh/suo/351/head -> origin/gh/suo/351/head 2022-03-17T19:04:58.8580449Z * [new branch] gh/suo/351/orig -> origin/gh/suo/351/orig 2022-03-17T19:04:58.8582176Z * [new branch] gh/suo/362/base -> origin/gh/suo/362/base 2022-03-17T19:04:58.8583607Z * [new branch] gh/suo/362/head -> origin/gh/suo/362/head 2022-03-17T19:04:58.8584941Z * [new branch] gh/suo/362/orig -> origin/gh/suo/362/orig 2022-03-17T19:04:58.8586677Z * [new branch] gh/suo/373/base -> origin/gh/suo/373/base 2022-03-17T19:04:58.8588080Z * [new branch] gh/suo/373/head -> origin/gh/suo/373/head 2022-03-17T19:04:58.8589398Z * [new branch] gh/suo/373/orig -> origin/gh/suo/373/orig 2022-03-17T19:04:58.8590861Z * [new branch] gh/suo/375/base -> origin/gh/suo/375/base 2022-03-17T19:04:58.8592192Z * [new branch] gh/suo/375/head -> origin/gh/suo/375/head 2022-03-17T19:04:58.8593231Z * [new branch] gh/suo/375/orig -> origin/gh/suo/375/orig 2022-03-17T19:04:58.8595064Z * [new branch] gh/suo/377/base -> origin/gh/suo/377/base 2022-03-17T19:04:58.8596487Z * [new branch] gh/suo/377/head -> origin/gh/suo/377/head 2022-03-17T19:04:58.8597909Z * [new branch] gh/suo/377/orig -> origin/gh/suo/377/orig 2022-03-17T19:04:58.8599707Z * [new branch] gh/suo/385/base -> origin/gh/suo/385/base 2022-03-17T19:04:58.8601256Z * [new branch] gh/suo/385/head -> origin/gh/suo/385/head 2022-03-17T19:04:58.8603006Z * [new branch] gh/suo/385/orig -> origin/gh/suo/385/orig 2022-03-17T19:04:58.8604745Z * [new branch] gh/suo/386/base -> origin/gh/suo/386/base 2022-03-17T19:04:58.8605883Z * [new branch] gh/suo/386/orig -> origin/gh/suo/386/orig 2022-03-17T19:04:58.8607668Z * [new branch] gh/suo/390/base -> origin/gh/suo/390/base 2022-03-17T19:04:58.8609048Z * [new branch] gh/suo/390/orig -> origin/gh/suo/390/orig 2022-03-17T19:04:58.8610738Z * [new branch] gh/suo/395/base -> origin/gh/suo/395/base 2022-03-17T19:04:58.8612012Z * [new branch] gh/suo/395/head -> origin/gh/suo/395/head 2022-03-17T19:04:58.8613338Z * [new branch] gh/suo/395/orig -> origin/gh/suo/395/orig 2022-03-17T19:04:58.8615031Z * [new branch] gh/suo/403/base -> origin/gh/suo/403/base 2022-03-17T19:04:58.8616371Z * [new branch] gh/suo/403/orig -> origin/gh/suo/403/orig 2022-03-17T19:04:58.8618090Z * [new branch] gh/suo/415/base -> origin/gh/suo/415/base 2022-03-17T19:04:58.8619324Z * [new branch] gh/suo/415/head -> origin/gh/suo/415/head 2022-03-17T19:04:58.8620726Z * [new branch] gh/suo/415/orig -> origin/gh/suo/415/orig 2022-03-17T19:04:58.8622461Z * [new branch] gh/suo/421/base -> origin/gh/suo/421/base 2022-03-17T19:04:58.8623983Z * [new branch] gh/suo/421/head -> origin/gh/suo/421/head 2022-03-17T19:04:58.8625492Z * [new branch] gh/suo/421/orig -> origin/gh/suo/421/orig 2022-03-17T19:04:58.8627204Z * [new branch] gh/suo/435/base -> origin/gh/suo/435/base 2022-03-17T19:04:58.8628652Z * [new branch] gh/suo/435/head -> origin/gh/suo/435/head 2022-03-17T19:04:58.8629775Z * [new branch] gh/suo/435/orig -> origin/gh/suo/435/orig 2022-03-17T19:04:58.8631551Z * [new branch] gh/suo/445/base -> origin/gh/suo/445/base 2022-03-17T19:04:58.8632622Z * [new branch] gh/suo/445/head -> origin/gh/suo/445/head 2022-03-17T19:04:58.8633901Z * [new branch] gh/suo/445/orig -> origin/gh/suo/445/orig 2022-03-17T19:04:58.8636032Z * [new branch] gh/suo/446/base -> origin/gh/suo/446/base 2022-03-17T19:04:58.8637019Z * [new branch] gh/suo/446/head -> origin/gh/suo/446/head 2022-03-17T19:04:58.8638412Z * [new branch] gh/suo/446/orig -> origin/gh/suo/446/orig 2022-03-17T19:04:58.8640241Z * [new branch] gh/suo/454/base -> origin/gh/suo/454/base 2022-03-17T19:04:58.8641582Z * [new branch] gh/suo/454/head -> origin/gh/suo/454/head 2022-03-17T19:04:58.8643649Z * [new branch] gh/suo/454/orig -> origin/gh/suo/454/orig 2022-03-17T19:04:58.8645447Z * [new branch] gh/suo/457/base -> origin/gh/suo/457/base 2022-03-17T19:04:58.8646777Z * [new branch] gh/suo/457/head -> origin/gh/suo/457/head 2022-03-17T19:04:58.8648065Z * [new branch] gh/suo/457/orig -> origin/gh/suo/457/orig 2022-03-17T19:04:58.8649894Z * [new branch] gh/suo/463/base -> origin/gh/suo/463/base 2022-03-17T19:04:58.8651340Z * [new branch] gh/suo/463/head -> origin/gh/suo/463/head 2022-03-17T19:04:58.8652669Z * [new branch] gh/suo/463/orig -> origin/gh/suo/463/orig 2022-03-17T19:04:58.8654125Z * [new branch] gh/suo/464/base -> origin/gh/suo/464/base 2022-03-17T19:04:58.8655581Z * [new branch] gh/suo/464/head -> origin/gh/suo/464/head 2022-03-17T19:04:58.8657088Z * [new branch] gh/suo/464/orig -> origin/gh/suo/464/orig 2022-03-17T19:04:58.8658584Z * [new branch] gh/suo/465/base -> origin/gh/suo/465/base 2022-03-17T19:04:58.8660075Z * [new branch] gh/suo/465/head -> origin/gh/suo/465/head 2022-03-17T19:04:58.8661083Z * [new branch] gh/suo/465/orig -> origin/gh/suo/465/orig 2022-03-17T19:04:58.8663010Z * [new branch] gh/suo/469/base -> origin/gh/suo/469/base 2022-03-17T19:04:58.8664372Z * [new branch] gh/suo/469/head -> origin/gh/suo/469/head 2022-03-17T19:04:58.8665676Z * [new branch] gh/suo/469/orig -> origin/gh/suo/469/orig 2022-03-17T19:04:58.8667348Z * [new branch] gh/suo/470/base -> origin/gh/suo/470/base 2022-03-17T19:04:58.8668688Z * [new branch] gh/suo/470/head -> origin/gh/suo/470/head 2022-03-17T19:04:58.8669948Z * [new branch] gh/suo/470/orig -> origin/gh/suo/470/orig 2022-03-17T19:04:58.8671679Z * [new branch] gh/suo/471/base -> origin/gh/suo/471/base 2022-03-17T19:04:58.8673157Z * [new branch] gh/suo/471/head -> origin/gh/suo/471/head 2022-03-17T19:04:58.8674372Z * [new branch] gh/suo/471/orig -> origin/gh/suo/471/orig 2022-03-17T19:04:58.8676130Z * [new branch] gh/suo/473/base -> origin/gh/suo/473/base 2022-03-17T19:04:58.8677429Z * [new branch] gh/suo/473/head -> origin/gh/suo/473/head 2022-03-17T19:04:58.8678789Z * [new branch] gh/suo/473/orig -> origin/gh/suo/473/orig 2022-03-17T19:04:58.8680449Z * [new branch] gh/suo/474/base -> origin/gh/suo/474/base 2022-03-17T19:04:58.8681759Z * [new branch] gh/suo/474/orig -> origin/gh/suo/474/orig 2022-03-17T19:04:58.8683732Z * [new branch] gh/suo/475/base -> origin/gh/suo/475/base 2022-03-17T19:04:58.8684867Z * [new branch] gh/suo/475/head -> origin/gh/suo/475/head 2022-03-17T19:04:58.8686344Z * [new branch] gh/suo/475/orig -> origin/gh/suo/475/orig 2022-03-17T19:04:58.8688104Z * [new branch] gh/suo/476/base -> origin/gh/suo/476/base 2022-03-17T19:04:58.8689228Z * [new branch] gh/suo/476/head -> origin/gh/suo/476/head 2022-03-17T19:04:58.8690738Z * [new branch] gh/suo/476/orig -> origin/gh/suo/476/orig 2022-03-17T19:04:58.8692352Z * [new branch] gh/suo/483/base -> origin/gh/suo/483/base 2022-03-17T19:04:58.8693663Z * [new branch] gh/suo/483/head -> origin/gh/suo/483/head 2022-03-17T19:04:58.8694999Z * [new branch] gh/suo/483/orig -> origin/gh/suo/483/orig 2022-03-17T19:04:58.8696791Z * [new branch] gh/suo/484/base -> origin/gh/suo/484/base 2022-03-17T19:04:58.8698107Z * [new branch] gh/suo/484/head -> origin/gh/suo/484/head 2022-03-17T19:04:58.8699533Z * [new branch] gh/suo/484/orig -> origin/gh/suo/484/orig 2022-03-17T19:04:58.8701196Z * [new branch] gh/suo/485/base -> origin/gh/suo/485/base 2022-03-17T19:04:58.8702470Z * [new branch] gh/suo/485/head -> origin/gh/suo/485/head 2022-03-17T19:04:58.8703799Z * [new branch] gh/suo/485/orig -> origin/gh/suo/485/orig 2022-03-17T19:04:58.8705643Z * [new branch] gh/suo/486/base -> origin/gh/suo/486/base 2022-03-17T19:04:58.8706944Z * [new branch] gh/suo/486/orig -> origin/gh/suo/486/orig 2022-03-17T19:04:58.8708564Z * [new branch] gh/suo/487/base -> origin/gh/suo/487/base 2022-03-17T19:04:58.8709935Z * [new branch] gh/suo/487/orig -> origin/gh/suo/487/orig 2022-03-17T19:04:58.8711695Z * [new branch] gh/suo/491/base -> origin/gh/suo/491/base 2022-03-17T19:04:58.8713085Z * [new branch] gh/suo/491/orig -> origin/gh/suo/491/orig 2022-03-17T19:04:58.8714910Z * [new branch] gh/suo/492/base -> origin/gh/suo/492/base 2022-03-17T19:04:58.8715927Z * [new branch] gh/suo/492/orig -> origin/gh/suo/492/orig 2022-03-17T19:04:58.8717739Z * [new branch] gh/suo/493/base -> origin/gh/suo/493/base 2022-03-17T19:04:58.8719087Z * [new branch] gh/suo/493/head -> origin/gh/suo/493/head 2022-03-17T19:04:58.8720405Z * [new branch] gh/suo/493/orig -> origin/gh/suo/493/orig 2022-03-17T19:04:58.8722236Z * [new branch] gh/suo/494/base -> origin/gh/suo/494/base 2022-03-17T19:04:58.8724172Z * [new branch] gh/suo/494/head -> origin/gh/suo/494/head 2022-03-17T19:04:58.8725451Z * [new branch] gh/suo/494/orig -> origin/gh/suo/494/orig 2022-03-17T19:04:58.8727182Z * [new branch] gh/suo/495/base -> origin/gh/suo/495/base 2022-03-17T19:04:58.8728472Z * [new branch] gh/suo/495/head -> origin/gh/suo/495/head 2022-03-17T19:04:58.8729754Z * [new branch] gh/suo/495/orig -> origin/gh/suo/495/orig 2022-03-17T19:04:58.8731521Z * [new branch] gh/suo/496/base -> origin/gh/suo/496/base 2022-03-17T19:04:58.8732934Z * [new branch] gh/suo/496/head -> origin/gh/suo/496/head 2022-03-17T19:04:58.8734241Z * [new branch] gh/suo/496/orig -> origin/gh/suo/496/orig 2022-03-17T19:04:58.8735885Z * [new branch] gh/suo/497/base -> origin/gh/suo/497/base 2022-03-17T19:04:58.8737189Z * [new branch] gh/suo/497/head -> origin/gh/suo/497/head 2022-03-17T19:04:58.8738556Z * [new branch] gh/suo/497/orig -> origin/gh/suo/497/orig 2022-03-17T19:04:58.8740345Z * [new branch] gh/suo/498/base -> origin/gh/suo/498/base 2022-03-17T19:04:58.8741745Z * [new branch] gh/suo/498/head -> origin/gh/suo/498/head 2022-03-17T19:04:58.8743161Z * [new branch] gh/suo/498/orig -> origin/gh/suo/498/orig 2022-03-17T19:04:58.8744991Z * [new branch] gh/suo/65/base -> origin/gh/suo/65/base 2022-03-17T19:04:58.8746610Z * [new branch] gh/suo/93/base -> origin/gh/suo/93/base 2022-03-17T19:04:58.8747900Z * [new branch] gh/suo/93/head -> origin/gh/suo/93/head 2022-03-17T19:04:58.8749928Z * [new branch] gh/supriyar/122/base -> origin/gh/supriyar/122/base 2022-03-17T19:04:58.8751389Z * [new branch] gh/supriyar/122/head -> origin/gh/supriyar/122/head 2022-03-17T19:04:58.8752569Z * [new branch] gh/supriyar/122/orig -> origin/gh/supriyar/122/orig 2022-03-17T19:04:58.8754400Z * [new branch] gh/supriyar/131/base -> origin/gh/supriyar/131/base 2022-03-17T19:04:58.8755768Z * [new branch] gh/supriyar/131/head -> origin/gh/supriyar/131/head 2022-03-17T19:04:58.8757103Z * [new branch] gh/supriyar/131/orig -> origin/gh/supriyar/131/orig 2022-03-17T19:04:58.8758823Z * [new branch] gh/supriyar/155/base -> origin/gh/supriyar/155/base 2022-03-17T19:04:58.8760310Z * [new branch] gh/supriyar/155/head -> origin/gh/supriyar/155/head 2022-03-17T19:04:58.8761759Z * [new branch] gh/supriyar/155/orig -> origin/gh/supriyar/155/orig 2022-03-17T19:04:58.8763859Z * [new branch] gh/supriyar/182/base -> origin/gh/supriyar/182/base 2022-03-17T19:04:58.8765224Z * [new branch] gh/supriyar/182/head -> origin/gh/supriyar/182/head 2022-03-17T19:04:58.8766917Z * [new branch] gh/supriyar/197/base -> origin/gh/supriyar/197/base 2022-03-17T19:04:58.8768103Z * [new branch] gh/supriyar/197/head -> origin/gh/supriyar/197/head 2022-03-17T19:04:58.8769606Z * [new branch] gh/supriyar/197/orig -> origin/gh/supriyar/197/orig 2022-03-17T19:04:58.8771243Z * [new branch] gh/supriyar/199/base -> origin/gh/supriyar/199/base 2022-03-17T19:04:58.8772395Z * [new branch] gh/supriyar/199/head -> origin/gh/supriyar/199/head 2022-03-17T19:04:58.8773865Z * [new branch] gh/supriyar/199/orig -> origin/gh/supriyar/199/orig 2022-03-17T19:04:58.8775629Z * [new branch] gh/supriyar/2/base -> origin/gh/supriyar/2/base 2022-03-17T19:04:58.8777399Z * [new branch] gh/supriyar/202/base -> origin/gh/supriyar/202/base 2022-03-17T19:04:58.8778753Z * [new branch] gh/supriyar/202/head -> origin/gh/supriyar/202/head 2022-03-17T19:04:58.8780162Z * [new branch] gh/supriyar/202/orig -> origin/gh/supriyar/202/orig 2022-03-17T19:04:58.8781780Z * [new branch] gh/supriyar/210/base -> origin/gh/supriyar/210/base 2022-03-17T19:04:58.8782905Z * [new branch] gh/supriyar/210/head -> origin/gh/supriyar/210/head 2022-03-17T19:04:58.8784224Z * [new branch] gh/supriyar/210/orig -> origin/gh/supriyar/210/orig 2022-03-17T19:04:58.8785897Z * [new branch] gh/supriyar/214/base -> origin/gh/supriyar/214/base 2022-03-17T19:04:58.8787658Z * [new branch] gh/supriyar/214/head -> origin/gh/supriyar/214/head 2022-03-17T19:04:58.8788761Z * [new branch] gh/supriyar/214/orig -> origin/gh/supriyar/214/orig 2022-03-17T19:04:58.8790723Z * [new branch] gh/supriyar/225/base -> origin/gh/supriyar/225/base 2022-03-17T19:04:58.8792330Z * [new branch] gh/supriyar/225/head -> origin/gh/supriyar/225/head 2022-03-17T19:04:58.8793777Z * [new branch] gh/supriyar/225/orig -> origin/gh/supriyar/225/orig 2022-03-17T19:04:58.8795354Z * [new branch] gh/supriyar/226/base -> origin/gh/supriyar/226/base 2022-03-17T19:04:58.8796462Z * [new branch] gh/supriyar/226/head -> origin/gh/supriyar/226/head 2022-03-17T19:04:58.8797847Z * [new branch] gh/supriyar/226/orig -> origin/gh/supriyar/226/orig 2022-03-17T19:04:58.8799535Z * [new branch] gh/supriyar/253/base -> origin/gh/supriyar/253/base 2022-03-17T19:04:58.8800965Z * [new branch] gh/supriyar/253/head -> origin/gh/supriyar/253/head 2022-03-17T19:04:58.8802403Z * [new branch] gh/supriyar/253/orig -> origin/gh/supriyar/253/orig 2022-03-17T19:04:58.8804889Z * [new branch] gh/supriyar/256/base -> origin/gh/supriyar/256/base 2022-03-17T19:04:58.8806066Z * [new branch] gh/supriyar/256/head -> origin/gh/supriyar/256/head 2022-03-17T19:04:58.8807543Z * [new branch] gh/supriyar/256/orig -> origin/gh/supriyar/256/orig 2022-03-17T19:04:58.8809413Z * [new branch] gh/supriyar/257/base -> origin/gh/supriyar/257/base 2022-03-17T19:04:58.8810576Z * [new branch] gh/supriyar/257/head -> origin/gh/supriyar/257/head 2022-03-17T19:04:58.8812246Z * [new branch] gh/supriyar/257/orig -> origin/gh/supriyar/257/orig 2022-03-17T19:04:58.8814134Z * [new branch] gh/supriyar/27/base -> origin/gh/supriyar/27/base 2022-03-17T19:04:58.8815464Z * [new branch] gh/supriyar/27/orig -> origin/gh/supriyar/27/orig 2022-03-17T19:04:58.8817263Z * [new branch] gh/supriyar/38/base -> origin/gh/supriyar/38/base 2022-03-17T19:04:58.8818618Z * [new branch] gh/supriyar/38/head -> origin/gh/supriyar/38/head 2022-03-17T19:04:58.8819937Z * [new branch] gh/supriyar/38/orig -> origin/gh/supriyar/38/orig 2022-03-17T19:04:58.8821664Z * [new branch] gh/supriyar/47/base -> origin/gh/supriyar/47/base 2022-03-17T19:04:58.8823081Z * [new branch] gh/supriyar/47/head -> origin/gh/supriyar/47/head 2022-03-17T19:04:58.8824422Z * [new branch] gh/supriyar/47/orig -> origin/gh/supriyar/47/orig 2022-03-17T19:04:58.8826316Z * [new branch] gh/supriyar/49/base -> origin/gh/supriyar/49/base 2022-03-17T19:04:58.8827434Z * [new branch] gh/supriyar/49/head -> origin/gh/supriyar/49/head 2022-03-17T19:04:58.8828920Z * [new branch] gh/supriyar/49/orig -> origin/gh/supriyar/49/orig 2022-03-17T19:04:58.8830790Z * [new branch] gh/supriyar/67/base -> origin/gh/supriyar/67/base 2022-03-17T19:04:58.8832196Z * [new branch] gh/supriyar/67/head -> origin/gh/supriyar/67/head 2022-03-17T19:04:58.8833587Z * [new branch] gh/supriyar/67/orig -> origin/gh/supriyar/67/orig 2022-03-17T19:04:58.8835069Z * [new branch] gh/supriyar/68/base -> origin/gh/supriyar/68/base 2022-03-17T19:04:58.8836378Z * [new branch] gh/supriyar/68/head -> origin/gh/supriyar/68/head 2022-03-17T19:04:58.8837793Z * [new branch] gh/supriyar/68/orig -> origin/gh/supriyar/68/orig 2022-03-17T19:04:58.8839630Z * [new branch] gh/supriyar/70/base -> origin/gh/supriyar/70/base 2022-03-17T19:04:58.8840952Z * [new branch] gh/supriyar/70/head -> origin/gh/supriyar/70/head 2022-03-17T19:04:58.8842179Z * [new branch] gh/supriyar/70/orig -> origin/gh/supriyar/70/orig 2022-03-17T19:04:58.8844072Z * [new branch] gh/supriyar/72/base -> origin/gh/supriyar/72/base 2022-03-17T19:04:58.8845150Z * [new branch] gh/supriyar/72/head -> origin/gh/supriyar/72/head 2022-03-17T19:04:58.8846462Z * [new branch] gh/supriyar/72/orig -> origin/gh/supriyar/72/orig 2022-03-17T19:04:58.8847987Z * [new branch] gh/supriyar/74/base -> origin/gh/supriyar/74/base 2022-03-17T19:04:58.8849436Z * [new branch] gh/supriyar/74/head -> origin/gh/supriyar/74/head 2022-03-17T19:04:58.8850891Z * [new branch] gh/supriyar/74/orig -> origin/gh/supriyar/74/orig 2022-03-17T19:04:58.8852435Z * [new branch] gh/supriyar/75/base -> origin/gh/supriyar/75/base 2022-03-17T19:04:58.8853753Z * [new branch] gh/supriyar/75/head -> origin/gh/supriyar/75/head 2022-03-17T19:04:58.8855105Z * [new branch] gh/supriyar/75/orig -> origin/gh/supriyar/75/orig 2022-03-17T19:04:58.8856816Z * [new branch] gh/supriyar/79/base -> origin/gh/supriyar/79/base 2022-03-17T19:04:58.8858194Z * [new branch] gh/supriyar/79/head -> origin/gh/supriyar/79/head 2022-03-17T19:04:58.8859608Z * [new branch] gh/supriyar/79/orig -> origin/gh/supriyar/79/orig 2022-03-17T19:04:58.8861477Z * [new branch] gh/supriyar/99/base -> origin/gh/supriyar/99/base 2022-03-17T19:04:58.8862817Z * [new branch] gh/supriyar/99/head -> origin/gh/supriyar/99/head 2022-03-17T19:04:58.8864154Z * [new branch] gh/supriyar/99/orig -> origin/gh/supriyar/99/orig 2022-03-17T19:04:58.8866105Z * [new branch] gh/swolchok/1/base -> origin/gh/swolchok/1/base 2022-03-17T19:04:58.8867445Z * [new branch] gh/swolchok/1/head -> origin/gh/swolchok/1/head 2022-03-17T19:04:58.8869198Z * [new branch] gh/swolchok/107/base -> origin/gh/swolchok/107/base 2022-03-17T19:04:58.8871859Z * [new branch] gh/swolchok/107/head -> origin/gh/swolchok/107/head 2022-03-17T19:04:58.8872679Z * [new branch] gh/swolchok/118/base -> origin/gh/swolchok/118/base 2022-03-17T19:04:58.8873759Z * [new branch] gh/swolchok/118/head -> origin/gh/swolchok/118/head 2022-03-17T19:04:58.8875180Z * [new branch] gh/swolchok/119/base -> origin/gh/swolchok/119/base 2022-03-17T19:04:58.8876525Z * [new branch] gh/swolchok/119/head -> origin/gh/swolchok/119/head 2022-03-17T19:04:58.8878894Z * [new branch] gh/swolchok/12/base -> origin/gh/swolchok/12/base 2022-03-17T19:04:58.8879907Z * [new branch] gh/swolchok/12/head -> origin/gh/swolchok/12/head 2022-03-17T19:04:58.8881266Z * [new branch] gh/swolchok/12/orig -> origin/gh/swolchok/12/orig 2022-03-17T19:04:58.8883619Z * [new branch] gh/swolchok/120/base -> origin/gh/swolchok/120/base 2022-03-17T19:04:58.8884688Z * [new branch] gh/swolchok/120/head -> origin/gh/swolchok/120/head 2022-03-17T19:04:58.8886309Z * [new branch] gh/swolchok/121/base -> origin/gh/swolchok/121/base 2022-03-17T19:04:58.8887469Z * [new branch] gh/swolchok/121/head -> origin/gh/swolchok/121/head 2022-03-17T19:04:58.8889134Z * [new branch] gh/swolchok/122/base -> origin/gh/swolchok/122/base 2022-03-17T19:04:58.8890214Z * [new branch] gh/swolchok/122/head -> origin/gh/swolchok/122/head 2022-03-17T19:04:58.8891894Z * [new branch] gh/swolchok/123/base -> origin/gh/swolchok/123/base 2022-03-17T19:04:58.8892963Z * [new branch] gh/swolchok/123/head -> origin/gh/swolchok/123/head 2022-03-17T19:04:58.8894798Z * [new branch] gh/swolchok/124/base -> origin/gh/swolchok/124/base 2022-03-17T19:04:58.8896043Z * [new branch] gh/swolchok/124/head -> origin/gh/swolchok/124/head 2022-03-17T19:04:58.8897601Z * [new branch] gh/swolchok/124/orig -> origin/gh/swolchok/124/orig 2022-03-17T19:04:58.8898957Z * [new branch] gh/swolchok/138/base -> origin/gh/swolchok/138/base 2022-03-17T19:04:58.8900605Z * [new branch] gh/swolchok/138/orig -> origin/gh/swolchok/138/orig 2022-03-17T19:04:58.8902338Z * [new branch] gh/swolchok/139/base -> origin/gh/swolchok/139/base 2022-03-17T19:04:58.8903500Z * [new branch] gh/swolchok/139/head -> origin/gh/swolchok/139/head 2022-03-17T19:04:58.8905028Z * [new branch] gh/swolchok/139/orig -> origin/gh/swolchok/139/orig 2022-03-17T19:04:58.8906731Z * [new branch] gh/swolchok/170/base -> origin/gh/swolchok/170/base 2022-03-17T19:04:58.8908078Z * [new branch] gh/swolchok/170/orig -> origin/gh/swolchok/170/orig 2022-03-17T19:04:58.8909911Z * [new branch] gh/swolchok/183/base -> origin/gh/swolchok/183/base 2022-03-17T19:04:58.8911205Z * [new branch] gh/swolchok/183/head -> origin/gh/swolchok/183/head 2022-03-17T19:04:58.8912542Z * [new branch] gh/swolchok/183/orig -> origin/gh/swolchok/183/orig 2022-03-17T19:04:58.8914357Z * [new branch] gh/swolchok/201/base -> origin/gh/swolchok/201/base 2022-03-17T19:04:58.8915573Z * [new branch] gh/swolchok/201/head -> origin/gh/swolchok/201/head 2022-03-17T19:04:58.8917180Z * [new branch] gh/swolchok/201/orig -> origin/gh/swolchok/201/orig 2022-03-17T19:04:58.8918745Z * [new branch] gh/swolchok/203/base -> origin/gh/swolchok/203/base 2022-03-17T19:04:58.8919960Z * [new branch] gh/swolchok/203/head -> origin/gh/swolchok/203/head 2022-03-17T19:04:58.8921545Z * [new branch] gh/swolchok/203/orig -> origin/gh/swolchok/203/orig 2022-03-17T19:04:58.8923361Z * [new branch] gh/swolchok/246/base -> origin/gh/swolchok/246/base 2022-03-17T19:04:58.8924569Z * [new branch] gh/swolchok/246/head -> origin/gh/swolchok/246/head 2022-03-17T19:04:58.8926292Z * [new branch] gh/swolchok/246/orig -> origin/gh/swolchok/246/orig 2022-03-17T19:04:58.8928035Z * [new branch] gh/swolchok/247/base -> origin/gh/swolchok/247/base 2022-03-17T19:04:58.8929252Z * [new branch] gh/swolchok/247/head -> origin/gh/swolchok/247/head 2022-03-17T19:04:58.8930591Z * [new branch] gh/swolchok/247/orig -> origin/gh/swolchok/247/orig 2022-03-17T19:04:58.8932543Z * [new branch] gh/swolchok/248/base -> origin/gh/swolchok/248/base 2022-03-17T19:04:58.8933605Z * [new branch] gh/swolchok/248/head -> origin/gh/swolchok/248/head 2022-03-17T19:04:58.8934892Z * [new branch] gh/swolchok/248/orig -> origin/gh/swolchok/248/orig 2022-03-17T19:04:58.8936788Z * [new branch] gh/swolchok/250/base -> origin/gh/swolchok/250/base 2022-03-17T19:04:58.8937932Z * [new branch] gh/swolchok/250/head -> origin/gh/swolchok/250/head 2022-03-17T19:04:58.8939440Z * [new branch] gh/swolchok/250/orig -> origin/gh/swolchok/250/orig 2022-03-17T19:04:58.8941191Z * [new branch] gh/swolchok/257/base -> origin/gh/swolchok/257/base 2022-03-17T19:04:58.8942381Z * [new branch] gh/swolchok/257/head -> origin/gh/swolchok/257/head 2022-03-17T19:04:58.8943924Z * [new branch] gh/swolchok/257/orig -> origin/gh/swolchok/257/orig 2022-03-17T19:04:58.8945616Z * [new branch] gh/swolchok/261/base -> origin/gh/swolchok/261/base 2022-03-17T19:04:58.8946912Z * [new branch] gh/swolchok/261/head -> origin/gh/swolchok/261/head 2022-03-17T19:04:58.8948178Z * [new branch] gh/swolchok/261/orig -> origin/gh/swolchok/261/orig 2022-03-17T19:04:58.8950096Z * [new branch] gh/swolchok/281/base -> origin/gh/swolchok/281/base 2022-03-17T19:04:58.8951287Z * [new branch] gh/swolchok/281/head -> origin/gh/swolchok/281/head 2022-03-17T19:04:58.8952880Z * [new branch] gh/swolchok/281/orig -> origin/gh/swolchok/281/orig 2022-03-17T19:04:58.8955311Z * [new branch] gh/swolchok/282/base -> origin/gh/swolchok/282/base 2022-03-17T19:04:58.8955949Z * [new branch] gh/swolchok/282/head -> origin/gh/swolchok/282/head 2022-03-17T19:04:58.8957635Z * [new branch] gh/swolchok/283/base -> origin/gh/swolchok/283/base 2022-03-17T19:04:58.8958708Z * [new branch] gh/swolchok/283/head -> origin/gh/swolchok/283/head 2022-03-17T19:04:58.8960776Z * [new branch] gh/swolchok/29/base -> origin/gh/swolchok/29/base 2022-03-17T19:04:58.8962109Z * [new branch] gh/swolchok/29/head -> origin/gh/swolchok/29/head 2022-03-17T19:04:58.8964474Z * [new branch] gh/swolchok/298/base -> origin/gh/swolchok/298/base 2022-03-17T19:04:58.8965625Z * [new branch] gh/swolchok/298/head -> origin/gh/swolchok/298/head 2022-03-17T19:04:58.8966949Z * [new branch] gh/swolchok/298/orig -> origin/gh/swolchok/298/orig 2022-03-17T19:04:58.8968796Z * [new branch] gh/swolchok/300/base -> origin/gh/swolchok/300/base 2022-03-17T19:04:58.8969959Z * [new branch] gh/swolchok/300/head -> origin/gh/swolchok/300/head 2022-03-17T19:04:58.8971486Z * [new branch] gh/swolchok/300/orig -> origin/gh/swolchok/300/orig 2022-03-17T19:04:58.8973265Z * [new branch] gh/swolchok/322/base -> origin/gh/swolchok/322/base 2022-03-17T19:04:58.8974463Z * [new branch] gh/swolchok/322/head -> origin/gh/swolchok/322/head 2022-03-17T19:04:58.8975780Z * [new branch] gh/swolchok/322/orig -> origin/gh/swolchok/322/orig 2022-03-17T19:04:58.8977673Z * [new branch] gh/swolchok/331/base -> origin/gh/swolchok/331/base 2022-03-17T19:04:58.8978795Z * [new branch] gh/swolchok/331/head -> origin/gh/swolchok/331/head 2022-03-17T19:04:58.8980354Z * [new branch] gh/swolchok/331/orig -> origin/gh/swolchok/331/orig 2022-03-17T19:04:58.8981994Z * [new branch] gh/swolchok/339/base -> origin/gh/swolchok/339/base 2022-03-17T19:04:58.8983262Z * [new branch] gh/swolchok/339/head -> origin/gh/swolchok/339/head 2022-03-17T19:04:58.8984937Z * [new branch] gh/swolchok/339/orig -> origin/gh/swolchok/339/orig 2022-03-17T19:04:58.8987096Z * [new branch] gh/swolchok/347/base -> origin/gh/swolchok/347/base 2022-03-17T19:04:58.8988155Z * [new branch] gh/swolchok/347/head -> origin/gh/swolchok/347/head 2022-03-17T19:04:58.8989440Z * [new branch] gh/swolchok/347/orig -> origin/gh/swolchok/347/orig 2022-03-17T19:04:58.8991079Z * [new branch] gh/swolchok/348/base -> origin/gh/swolchok/348/base 2022-03-17T19:04:58.8992851Z * [new branch] gh/swolchok/348/head -> origin/gh/swolchok/348/head 2022-03-17T19:04:58.8994420Z * [new branch] gh/swolchok/349/base -> origin/gh/swolchok/349/base 2022-03-17T19:04:58.8995695Z * [new branch] gh/swolchok/349/head -> origin/gh/swolchok/349/head 2022-03-17T19:04:58.8997311Z * [new branch] gh/swolchok/350/base -> origin/gh/swolchok/350/base 2022-03-17T19:04:58.8998913Z * [new branch] gh/swolchok/350/head -> origin/gh/swolchok/350/head 2022-03-17T19:04:58.9000140Z * [new branch] gh/swolchok/351/base -> origin/gh/swolchok/351/base 2022-03-17T19:04:58.9001795Z * [new branch] gh/swolchok/351/head -> origin/gh/swolchok/351/head 2022-03-17T19:04:58.9003594Z * [new branch] gh/swolchok/352/base -> origin/gh/swolchok/352/base 2022-03-17T19:04:58.9005071Z * [new branch] gh/swolchok/352/head -> origin/gh/swolchok/352/head 2022-03-17T19:04:58.9006606Z * [new branch] gh/swolchok/353/base -> origin/gh/swolchok/353/base 2022-03-17T19:04:58.9008000Z * [new branch] gh/swolchok/353/head -> origin/gh/swolchok/353/head 2022-03-17T19:04:58.9009719Z * [new branch] gh/swolchok/36/base -> origin/gh/swolchok/36/base 2022-03-17T19:04:58.9011120Z * [new branch] gh/swolchok/36/head -> origin/gh/swolchok/36/head 2022-03-17T19:04:58.9012990Z * [new branch] gh/swolchok/379/base -> origin/gh/swolchok/379/base 2022-03-17T19:04:58.9014121Z * [new branch] gh/swolchok/379/head -> origin/gh/swolchok/379/head 2022-03-17T19:04:58.9015589Z * [new branch] gh/swolchok/379/orig -> origin/gh/swolchok/379/orig 2022-03-17T19:04:58.9017151Z * [new branch] gh/swolchok/38/base -> origin/gh/swolchok/38/base 2022-03-17T19:04:58.9018613Z * [new branch] gh/swolchok/38/head -> origin/gh/swolchok/38/head 2022-03-17T19:04:58.9019972Z * [new branch] gh/swolchok/38/orig -> origin/gh/swolchok/38/orig 2022-03-17T19:04:58.9021775Z * [new branch] gh/swolchok/387/base -> origin/gh/swolchok/387/base 2022-03-17T19:04:58.9023146Z * [new branch] gh/swolchok/387/head -> origin/gh/swolchok/387/head 2022-03-17T19:04:58.9024530Z * [new branch] gh/swolchok/387/orig -> origin/gh/swolchok/387/orig 2022-03-17T19:04:58.9025978Z * [new branch] gh/swolchok/39/base -> origin/gh/swolchok/39/base 2022-03-17T19:04:58.9027474Z * [new branch] gh/swolchok/39/head -> origin/gh/swolchok/39/head 2022-03-17T19:04:58.9028865Z * [new branch] gh/swolchok/39/orig -> origin/gh/swolchok/39/orig 2022-03-17T19:04:58.9030617Z * [new branch] gh/swolchok/398/base -> origin/gh/swolchok/398/base 2022-03-17T19:04:58.9031996Z * [new branch] gh/swolchok/398/head -> origin/gh/swolchok/398/head 2022-03-17T19:04:58.9033340Z * [new branch] gh/swolchok/398/orig -> origin/gh/swolchok/398/orig 2022-03-17T19:04:58.9034833Z * [new branch] gh/swolchok/40/base -> origin/gh/swolchok/40/base 2022-03-17T19:04:58.9036242Z * [new branch] gh/swolchok/40/head -> origin/gh/swolchok/40/head 2022-03-17T19:04:58.9037681Z * [new branch] gh/swolchok/40/orig -> origin/gh/swolchok/40/orig 2022-03-17T19:04:58.9039572Z * [new branch] gh/swolchok/421/base -> origin/gh/swolchok/421/base 2022-03-17T19:04:58.9040594Z * [new branch] gh/swolchok/421/head -> origin/gh/swolchok/421/head 2022-03-17T19:04:58.9042128Z * [new branch] gh/swolchok/421/orig -> origin/gh/swolchok/421/orig 2022-03-17T19:04:58.9044372Z * [new branch] gh/swolchok/425/base -> origin/gh/swolchok/425/base 2022-03-17T19:04:58.9045473Z * [new branch] gh/swolchok/425/head -> origin/gh/swolchok/425/head 2022-03-17T19:04:58.9047456Z * [new branch] gh/swolchok/427/base -> origin/gh/swolchok/427/base 2022-03-17T19:04:58.9048819Z * [new branch] gh/swolchok/427/head -> origin/gh/swolchok/427/head 2022-03-17T19:04:58.9049981Z * [new branch] gh/swolchok/427/orig -> origin/gh/swolchok/427/orig 2022-03-17T19:04:58.9051865Z * [new branch] gh/swolchok/441/base -> origin/gh/swolchok/441/base 2022-03-17T19:04:58.9053221Z * [new branch] gh/swolchok/441/head -> origin/gh/swolchok/441/head 2022-03-17T19:04:58.9054519Z * [new branch] gh/swolchok/441/orig -> origin/gh/swolchok/441/orig 2022-03-17T19:04:58.9056417Z * [new branch] gh/swolchok/442/base -> origin/gh/swolchok/442/base 2022-03-17T19:04:58.9057828Z * [new branch] gh/swolchok/442/head -> origin/gh/swolchok/442/head 2022-03-17T19:04:58.9059261Z * [new branch] gh/swolchok/442/orig -> origin/gh/swolchok/442/orig 2022-03-17T19:04:58.9061065Z * [new branch] gh/swolchok/454/base -> origin/gh/swolchok/454/base 2022-03-17T19:04:58.9062228Z * [new branch] gh/swolchok/454/head -> origin/gh/swolchok/454/head 2022-03-17T19:04:58.9063789Z * [new branch] gh/swolchok/454/orig -> origin/gh/swolchok/454/orig 2022-03-17T19:04:58.9065506Z * [new branch] gh/swolchok/455/base -> origin/gh/swolchok/455/base 2022-03-17T19:04:58.9066841Z * [new branch] gh/swolchok/455/head -> origin/gh/swolchok/455/head 2022-03-17T19:04:58.9068193Z * [new branch] gh/swolchok/455/orig -> origin/gh/swolchok/455/orig 2022-03-17T19:04:58.9069926Z * [new branch] gh/swolchok/457/base -> origin/gh/swolchok/457/base 2022-03-17T19:04:58.9071070Z * [new branch] gh/swolchok/457/head -> origin/gh/swolchok/457/head 2022-03-17T19:04:58.9072574Z * [new branch] gh/swolchok/457/orig -> origin/gh/swolchok/457/orig 2022-03-17T19:04:58.9074345Z * [new branch] gh/swolchok/458/base -> origin/gh/swolchok/458/base 2022-03-17T19:04:58.9075653Z * [new branch] gh/swolchok/458/head -> origin/gh/swolchok/458/head 2022-03-17T19:04:58.9077107Z * [new branch] gh/swolchok/458/orig -> origin/gh/swolchok/458/orig 2022-03-17T19:04:58.9078803Z * [new branch] gh/swolchok/461/base -> origin/gh/swolchok/461/base 2022-03-17T19:04:58.9080167Z * [new branch] gh/swolchok/461/head -> origin/gh/swolchok/461/head 2022-03-17T19:04:58.9081489Z * [new branch] gh/swolchok/461/orig -> origin/gh/swolchok/461/orig 2022-03-17T19:04:58.9083451Z * [new branch] gh/swolchok/462/base -> origin/gh/swolchok/462/base 2022-03-17T19:04:58.9084830Z * [new branch] gh/swolchok/462/head -> origin/gh/swolchok/462/head 2022-03-17T19:04:58.9086176Z * [new branch] gh/swolchok/462/orig -> origin/gh/swolchok/462/orig 2022-03-17T19:04:58.9087966Z * [new branch] gh/swolchok/464/base -> origin/gh/swolchok/464/base 2022-03-17T19:04:58.9089454Z * [new branch] gh/swolchok/464/head -> origin/gh/swolchok/464/head 2022-03-17T19:04:58.9090499Z * [new branch] gh/swolchok/464/orig -> origin/gh/swolchok/464/orig 2022-03-17T19:04:58.9092288Z * [new branch] gh/swolchok/468/base -> origin/gh/swolchok/468/base 2022-03-17T19:04:58.9093907Z * [new branch] gh/swolchok/468/head -> origin/gh/swolchok/468/head 2022-03-17T19:04:58.9095049Z * [new branch] gh/swolchok/468/orig -> origin/gh/swolchok/468/orig 2022-03-17T19:04:58.9096994Z * [new branch] gh/swolchok/469/base -> origin/gh/swolchok/469/base 2022-03-17T19:04:58.9098371Z * [new branch] gh/swolchok/469/head -> origin/gh/swolchok/469/head 2022-03-17T19:04:58.9099721Z * [new branch] gh/swolchok/469/orig -> origin/gh/swolchok/469/orig 2022-03-17T19:04:58.9101567Z * [new branch] gh/swolchok/470/base -> origin/gh/swolchok/470/base 2022-03-17T19:04:58.9102895Z * [new branch] gh/swolchok/470/head -> origin/gh/swolchok/470/head 2022-03-17T19:04:58.9104243Z * [new branch] gh/swolchok/470/orig -> origin/gh/swolchok/470/orig 2022-03-17T19:04:58.9105939Z * [new branch] gh/swolchok/471/base -> origin/gh/swolchok/471/base 2022-03-17T19:04:58.9107363Z * [new branch] gh/swolchok/471/head -> origin/gh/swolchok/471/head 2022-03-17T19:04:58.9109000Z * [new branch] gh/swolchok/471/orig -> origin/gh/swolchok/471/orig 2022-03-17T19:04:58.9110323Z * [new branch] gh/swolchok/59/base -> origin/gh/swolchok/59/base 2022-03-17T19:04:58.9111743Z * [new branch] gh/swolchok/59/head -> origin/gh/swolchok/59/head 2022-03-17T19:04:58.9113482Z * [new branch] gh/swolchok/65/base -> origin/gh/swolchok/65/base 2022-03-17T19:04:58.9114804Z * [new branch] gh/swolchok/65/head -> origin/gh/swolchok/65/head 2022-03-17T19:04:58.9115943Z * [new branch] gh/swolchok/65/orig -> origin/gh/swolchok/65/orig 2022-03-17T19:04:58.9117973Z * [new branch] gh/swolchok/69/base -> origin/gh/swolchok/69/base 2022-03-17T19:04:58.9119396Z * [new branch] gh/swolchok/69/head -> origin/gh/swolchok/69/head 2022-03-17T19:04:58.9120853Z * [new branch] gh/swolchok/69/orig -> origin/gh/swolchok/69/orig 2022-03-17T19:04:58.9122660Z * [new branch] gh/swolchok/7/base -> origin/gh/swolchok/7/base 2022-03-17T19:04:58.9124147Z * [new branch] gh/swolchok/7/head -> origin/gh/swolchok/7/head 2022-03-17T19:04:58.9125998Z * [new branch] gh/swolchok/70/base -> origin/gh/swolchok/70/base 2022-03-17T19:04:58.9127189Z * [new branch] gh/swolchok/70/head -> origin/gh/swolchok/70/head 2022-03-17T19:04:58.9128653Z * [new branch] gh/swolchok/70/orig -> origin/gh/swolchok/70/orig 2022-03-17T19:04:58.9130173Z * [new branch] gh/swolchok/71/base -> origin/gh/swolchok/71/base 2022-03-17T19:04:58.9131540Z * [new branch] gh/swolchok/71/head -> origin/gh/swolchok/71/head 2022-03-17T19:04:58.9132717Z * [new branch] gh/swolchok/71/orig -> origin/gh/swolchok/71/orig 2022-03-17T19:04:58.9134613Z * [new branch] gh/swolchok/77/base -> origin/gh/swolchok/77/base 2022-03-17T19:04:58.9135991Z * [new branch] gh/swolchok/77/head -> origin/gh/swolchok/77/head 2022-03-17T19:04:58.9137209Z * [new branch] gh/swolchok/77/orig -> origin/gh/swolchok/77/orig 2022-03-17T19:04:58.9138886Z * [new branch] gh/swolchok/8/base -> origin/gh/swolchok/8/base 2022-03-17T19:04:58.9140377Z * [new branch] gh/swolchok/8/head -> origin/gh/swolchok/8/head 2022-03-17T19:04:58.9142242Z * [new branch] gh/syed-ahmed/4/base -> origin/gh/syed-ahmed/4/base 2022-03-17T19:04:58.9143936Z * [new branch] gh/syed-ahmed/5/base -> origin/gh/syed-ahmed/5/base 2022-03-17T19:04:58.9145667Z * [new branch] gh/syed-ahmed/7/base -> origin/gh/syed-ahmed/7/base 2022-03-17T19:04:58.9147612Z * [new branch] gh/taewookoh/2/base -> origin/gh/taewookoh/2/base 2022-03-17T19:04:58.9149182Z * [new branch] gh/taewookoh/2/head -> origin/gh/taewookoh/2/head 2022-03-17T19:04:58.9151036Z * [new branch] gh/taivu1998/10/base -> origin/gh/taivu1998/10/base 2022-03-17T19:04:58.9152449Z * [new branch] gh/taivu1998/10/head -> origin/gh/taivu1998/10/head 2022-03-17T19:04:58.9153838Z * [new branch] gh/taivu1998/10/orig -> origin/gh/taivu1998/10/orig 2022-03-17T19:04:58.9155310Z * [new branch] gh/taivu1998/11/base -> origin/gh/taivu1998/11/base 2022-03-17T19:04:58.9156682Z * [new branch] gh/taivu1998/11/head -> origin/gh/taivu1998/11/head 2022-03-17T19:04:58.9158019Z * [new branch] gh/taivu1998/11/orig -> origin/gh/taivu1998/11/orig 2022-03-17T19:04:58.9159720Z * [new branch] gh/taivu1998/12/base -> origin/gh/taivu1998/12/base 2022-03-17T19:04:58.9161214Z * [new branch] gh/taivu1998/12/head -> origin/gh/taivu1998/12/head 2022-03-17T19:04:58.9162837Z * [new branch] gh/taivu1998/12/orig -> origin/gh/taivu1998/12/orig 2022-03-17T19:04:58.9164575Z * [new branch] gh/taivu1998/13/base -> origin/gh/taivu1998/13/base 2022-03-17T19:04:58.9165795Z * [new branch] gh/taivu1998/13/head -> origin/gh/taivu1998/13/head 2022-03-17T19:04:58.9168443Z * [new branch] gh/taivu1998/13/orig -> origin/gh/taivu1998/13/orig 2022-03-17T19:04:58.9169455Z * [new branch] gh/taivu1998/7/base -> origin/gh/taivu1998/7/base 2022-03-17T19:04:58.9170690Z * [new branch] gh/taivu1998/7/head -> origin/gh/taivu1998/7/head 2022-03-17T19:04:58.9172101Z * [new branch] gh/taivu1998/7/orig -> origin/gh/taivu1998/7/orig 2022-03-17T19:04:58.9173686Z * [new branch] gh/taivu1998/8/base -> origin/gh/taivu1998/8/base 2022-03-17T19:04:58.9174978Z * [new branch] gh/taivu1998/8/head -> origin/gh/taivu1998/8/head 2022-03-17T19:04:58.9176441Z * [new branch] gh/taivu1998/8/orig -> origin/gh/taivu1998/8/orig 2022-03-17T19:04:58.9178025Z * [new branch] gh/taivu1998/9/base -> origin/gh/taivu1998/9/base 2022-03-17T19:04:58.9179179Z * [new branch] gh/taivu1998/9/head -> origin/gh/taivu1998/9/head 2022-03-17T19:04:58.9180621Z * [new branch] gh/taivu1998/9/orig -> origin/gh/taivu1998/9/orig 2022-03-17T19:04:58.9182926Z * [new branch] gh/taylorrobie/autotune_prototype -> origin/gh/taylorrobie/autotune_prototype 2022-03-17T19:04:58.9184367Z * [new branch] gh/taylorrobie/benchmark_utils_dataclassification -> origin/gh/taylorrobie/benchmark_utils_dataclassification 2022-03-17T19:04:58.9185451Z * [new branch] gh/taylorrobie/callgrind_backtest -> origin/gh/taylorrobie/callgrind_backtest 2022-03-17T19:04:58.9186797Z * [new branch] gh/taylorrobie/callgrind_scribe2 -> origin/gh/taylorrobie/callgrind_scribe2 2022-03-17T19:04:58.9187998Z * [new branch] gh/taylorrobie/composable_timer -> origin/gh/taylorrobie/composable_timer 2022-03-17T19:04:58.9189435Z * [new branch] gh/taylorrobie/fuzzed_op_microbenchmarks -> origin/gh/taylorrobie/fuzzed_op_microbenchmarks 2022-03-17T19:04:58.9190688Z * [new branch] gh/taylorrobie/kernel_overhead -> origin/gh/taylorrobie/kernel_overhead 2022-03-17T19:04:58.9191981Z * [new branch] gh/taylorrobie/unify_index -> origin/gh/taylorrobie/unify_index 2022-03-17T19:04:58.9193298Z * [new branch] gh/taylorrobie/wall_backtest -> origin/gh/taylorrobie/wall_backtest 2022-03-17T19:04:58.9195608Z * [new branch] gh/terrychenism/12/base -> origin/gh/terrychenism/12/base 2022-03-17T19:04:58.9196778Z * [new branch] gh/terrychenism/12/head -> origin/gh/terrychenism/12/head 2022-03-17T19:04:58.9198049Z * [new branch] gh/terrychenism/12/orig -> origin/gh/terrychenism/12/orig 2022-03-17T19:04:58.9200285Z * [new branch] gh/terrychenism/13/base -> origin/gh/terrychenism/13/base 2022-03-17T19:04:58.9201316Z * [new branch] gh/terrychenism/13/head -> origin/gh/terrychenism/13/head 2022-03-17T19:04:58.9202790Z * [new branch] gh/terrychenism/13/orig -> origin/gh/terrychenism/13/orig 2022-03-17T19:04:58.9205021Z * [new branch] gh/terrychenism/20/base -> origin/gh/terrychenism/20/base 2022-03-17T19:04:58.9206230Z * [new branch] gh/terrychenism/20/head -> origin/gh/terrychenism/20/head 2022-03-17T19:04:58.9207581Z * [new branch] gh/terrychenism/20/orig -> origin/gh/terrychenism/20/orig 2022-03-17T19:04:58.9209608Z * [new branch] gh/terrychenism/21/base -> origin/gh/terrychenism/21/base 2022-03-17T19:04:58.9210856Z * [new branch] gh/terrychenism/21/head -> origin/gh/terrychenism/21/head 2022-03-17T19:04:58.9212147Z * [new branch] gh/terrychenism/21/orig -> origin/gh/terrychenism/21/orig 2022-03-17T19:04:58.9213931Z * [new branch] gh/terrychenism/22/base -> origin/gh/terrychenism/22/base 2022-03-17T19:04:58.9215380Z * [new branch] gh/terrychenism/22/head -> origin/gh/terrychenism/22/head 2022-03-17T19:04:58.9216756Z * [new branch] gh/terrychenism/22/orig -> origin/gh/terrychenism/22/orig 2022-03-17T19:04:58.9218765Z * [new branch] gh/tierex/1/base -> origin/gh/tierex/1/base 2022-03-17T19:04:58.9220041Z * [new branch] gh/tierex/1/head -> origin/gh/tierex/1/head 2022-03-17T19:04:58.9221393Z * [new branch] gh/tierex/1/orig -> origin/gh/tierex/1/orig 2022-03-17T19:04:58.9223349Z * [new branch] gh/tierex/4/base -> origin/gh/tierex/4/base 2022-03-17T19:04:58.9224540Z * [new branch] gh/tierex/4/head -> origin/gh/tierex/4/head 2022-03-17T19:04:58.9225881Z * [new branch] gh/tierex/4/orig -> origin/gh/tierex/4/orig 2022-03-17T19:04:58.9227943Z * [new branch] gh/tierex/5/base -> origin/gh/tierex/5/base 2022-03-17T19:04:58.9229227Z * [new branch] gh/tierex/5/head -> origin/gh/tierex/5/head 2022-03-17T19:04:58.9230554Z * [new branch] gh/tierex/5/orig -> origin/gh/tierex/5/orig 2022-03-17T19:04:58.9232029Z * [new branch] gh/tierex/6/base -> origin/gh/tierex/6/base 2022-03-17T19:04:58.9233383Z * [new branch] gh/tierex/6/head -> origin/gh/tierex/6/head 2022-03-17T19:04:58.9234719Z * [new branch] gh/tierex/6/orig -> origin/gh/tierex/6/orig 2022-03-17T19:04:58.9236888Z * [new branch] gh/tktrungna/14/base -> origin/gh/tktrungna/14/base 2022-03-17T19:04:58.9238051Z * [new branch] gh/tktrungna/14/head -> origin/gh/tktrungna/14/head 2022-03-17T19:04:58.9239308Z * [new branch] gh/tktrungna/14/orig -> origin/gh/tktrungna/14/orig 2022-03-17T19:04:58.9240856Z * [new branch] gh/tktrungna/15/base -> origin/gh/tktrungna/15/base 2022-03-17T19:04:58.9242255Z * [new branch] gh/tktrungna/15/head -> origin/gh/tktrungna/15/head 2022-03-17T19:04:58.9243682Z * [new branch] gh/tktrungna/15/orig -> origin/gh/tktrungna/15/orig 2022-03-17T19:04:58.9245307Z * [new branch] gh/tktrungna/16/base -> origin/gh/tktrungna/16/base 2022-03-17T19:04:58.9246626Z * [new branch] gh/tktrungna/16/head -> origin/gh/tktrungna/16/head 2022-03-17T19:04:58.9247963Z * [new branch] gh/tktrungna/16/orig -> origin/gh/tktrungna/16/orig 2022-03-17T19:04:58.9249567Z * [new branch] gh/tktrungna/17/base -> origin/gh/tktrungna/17/base 2022-03-17T19:04:58.9250942Z * [new branch] gh/tktrungna/17/head -> origin/gh/tktrungna/17/head 2022-03-17T19:04:58.9252178Z * [new branch] gh/tktrungna/17/orig -> origin/gh/tktrungna/17/orig 2022-03-17T19:04:58.9254662Z * [new branch] gh/tktrungna/18/base -> origin/gh/tktrungna/18/base 2022-03-17T19:04:58.9255415Z * [new branch] gh/tktrungna/18/head -> origin/gh/tktrungna/18/head 2022-03-17T19:04:58.9256592Z * [new branch] gh/tktrungna/18/orig -> origin/gh/tktrungna/18/orig 2022-03-17T19:04:58.9258418Z * [new branch] gh/tktrungna/19/base -> origin/gh/tktrungna/19/base 2022-03-17T19:04:58.9259622Z * [new branch] gh/tktrungna/19/head -> origin/gh/tktrungna/19/head 2022-03-17T19:04:58.9261199Z * [new branch] gh/tktrungna/19/orig -> origin/gh/tktrungna/19/orig 2022-03-17T19:04:58.9263244Z * [new branch] gh/tugsbayasgalan/1/base -> origin/gh/tugsbayasgalan/1/base 2022-03-17T19:04:58.9264424Z * [new branch] gh/tugsbayasgalan/1/head -> origin/gh/tugsbayasgalan/1/head 2022-03-17T19:04:58.9266336Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2022-03-17T19:04:58.9267628Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2022-03-17T19:04:58.9269460Z * [new branch] gh/tugsbayasgalan/16/base -> origin/gh/tugsbayasgalan/16/base 2022-03-17T19:04:58.9270877Z * [new branch] gh/tugsbayasgalan/16/head -> origin/gh/tugsbayasgalan/16/head 2022-03-17T19:04:58.9272126Z * [new branch] gh/tugsbayasgalan/16/orig -> origin/gh/tugsbayasgalan/16/orig 2022-03-17T19:04:58.9273969Z * [new branch] gh/tugsbayasgalan/19/base -> origin/gh/tugsbayasgalan/19/base 2022-03-17T19:04:58.9275150Z * [new branch] gh/tugsbayasgalan/19/head -> origin/gh/tugsbayasgalan/19/head 2022-03-17T19:04:58.9276594Z * [new branch] gh/tugsbayasgalan/19/orig -> origin/gh/tugsbayasgalan/19/orig 2022-03-17T19:04:58.9278185Z * [new branch] gh/tugsbayasgalan/20/base -> origin/gh/tugsbayasgalan/20/base 2022-03-17T19:04:58.9279661Z * [new branch] gh/tugsbayasgalan/20/head -> origin/gh/tugsbayasgalan/20/head 2022-03-17T19:04:58.9280844Z * [new branch] gh/tugsbayasgalan/20/orig -> origin/gh/tugsbayasgalan/20/orig 2022-03-17T19:04:58.9286330Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2022-03-17T19:04:58.9287190Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2022-03-17T19:04:58.9287998Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2022-03-17T19:04:58.9288810Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2022-03-17T19:04:58.9289608Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2022-03-17T19:04:58.9290630Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2022-03-17T19:04:58.9292252Z * [new branch] gh/tugsbayasgalan/42/base -> origin/gh/tugsbayasgalan/42/base 2022-03-17T19:04:58.9293620Z * [new branch] gh/tugsbayasgalan/42/head -> origin/gh/tugsbayasgalan/42/head 2022-03-17T19:04:58.9294995Z * [new branch] gh/tugsbayasgalan/42/orig -> origin/gh/tugsbayasgalan/42/orig 2022-03-17T19:04:58.9296968Z * [new branch] gh/tugsbayasgalan/47/base -> origin/gh/tugsbayasgalan/47/base 2022-03-17T19:04:58.9298106Z * [new branch] gh/tugsbayasgalan/47/head -> origin/gh/tugsbayasgalan/47/head 2022-03-17T19:04:58.9299422Z * [new branch] gh/tugsbayasgalan/47/orig -> origin/gh/tugsbayasgalan/47/orig 2022-03-17T19:04:58.9300904Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2022-03-17T19:04:58.9302225Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2022-03-17T19:04:58.9303715Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2022-03-17T19:04:58.9305514Z * [new branch] gh/tugsbayasgalan/49/base -> origin/gh/tugsbayasgalan/49/base 2022-03-17T19:04:58.9306633Z * [new branch] gh/tugsbayasgalan/49/head -> origin/gh/tugsbayasgalan/49/head 2022-03-17T19:04:58.9308270Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2022-03-17T19:04:58.9309579Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2022-03-17T19:04:58.9310886Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2022-03-17T19:04:58.9313117Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2022-03-17T19:04:58.9314285Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2022-03-17T19:04:58.9315636Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2022-03-17T19:04:58.9317700Z * [new branch] gh/tugsbayasgalan/54/base -> origin/gh/tugsbayasgalan/54/base 2022-03-17T19:04:58.9318938Z * [new branch] gh/tugsbayasgalan/54/head -> origin/gh/tugsbayasgalan/54/head 2022-03-17T19:04:58.9320183Z * [new branch] gh/tugsbayasgalan/54/orig -> origin/gh/tugsbayasgalan/54/orig 2022-03-17T19:04:58.9322275Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2022-03-17T19:04:58.9323575Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2022-03-17T19:04:58.9324886Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2022-03-17T19:04:58.9327045Z * [new branch] gh/tugsbayasgalan/56/base -> origin/gh/tugsbayasgalan/56/base 2022-03-17T19:04:58.9328118Z * [new branch] gh/tugsbayasgalan/56/head -> origin/gh/tugsbayasgalan/56/head 2022-03-17T19:04:58.9329478Z * [new branch] gh/tugsbayasgalan/56/orig -> origin/gh/tugsbayasgalan/56/orig 2022-03-17T19:04:58.9331491Z * [new branch] gh/tugsbayasgalan/57/base -> origin/gh/tugsbayasgalan/57/base 2022-03-17T19:04:58.9332487Z * [new branch] gh/tugsbayasgalan/57/head -> origin/gh/tugsbayasgalan/57/head 2022-03-17T19:04:58.9333764Z * [new branch] gh/tugsbayasgalan/57/orig -> origin/gh/tugsbayasgalan/57/orig 2022-03-17T19:04:58.9335587Z * [new branch] gh/tugsbayasgalan/58/base -> origin/gh/tugsbayasgalan/58/base 2022-03-17T19:04:58.9336761Z * [new branch] gh/tugsbayasgalan/58/head -> origin/gh/tugsbayasgalan/58/head 2022-03-17T19:04:58.9338680Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2022-03-17T19:04:58.9339760Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2022-03-17T19:04:58.9341072Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2022-03-17T19:04:58.9343332Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2022-03-17T19:04:58.9344484Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2022-03-17T19:04:58.9345804Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2022-03-17T19:04:58.9347708Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2022-03-17T19:04:58.9348997Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2022-03-17T19:04:58.9350334Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2022-03-17T19:04:58.9352205Z * [new branch] gh/tugsbayasgalan/62/base -> origin/gh/tugsbayasgalan/62/base 2022-03-17T19:04:58.9353406Z * [new branch] gh/tugsbayasgalan/62/head -> origin/gh/tugsbayasgalan/62/head 2022-03-17T19:04:58.9354711Z * [new branch] gh/tugsbayasgalan/62/orig -> origin/gh/tugsbayasgalan/62/orig 2022-03-17T19:04:58.9356831Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2022-03-17T19:04:58.9357851Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2022-03-17T19:04:58.9359177Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2022-03-17T19:04:58.9361045Z * [new branch] gh/tugsbayasgalan/64/base -> origin/gh/tugsbayasgalan/64/base 2022-03-17T19:04:58.9362409Z * [new branch] gh/tugsbayasgalan/64/head -> origin/gh/tugsbayasgalan/64/head 2022-03-17T19:04:58.9363847Z * [new branch] gh/tugsbayasgalan/64/orig -> origin/gh/tugsbayasgalan/64/orig 2022-03-17T19:04:58.9365756Z * [new branch] gh/tugsbayasgalan/65/base -> origin/gh/tugsbayasgalan/65/base 2022-03-17T19:04:58.9366875Z * [new branch] gh/tugsbayasgalan/65/head -> origin/gh/tugsbayasgalan/65/head 2022-03-17T19:04:58.9368155Z * [new branch] gh/tugsbayasgalan/65/orig -> origin/gh/tugsbayasgalan/65/orig 2022-03-17T19:04:58.9370216Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2022-03-17T19:04:58.9371533Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2022-03-17T19:04:58.9372827Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2022-03-17T19:04:58.9376210Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2022-03-17T19:04:58.9376588Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2022-03-17T19:04:58.9377130Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2022-03-17T19:04:58.9379101Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2022-03-17T19:04:58.9380439Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2022-03-17T19:04:58.9381876Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2022-03-17T19:04:58.9383652Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2022-03-17T19:04:58.9385376Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2022-03-17T19:04:58.9386471Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2022-03-17T19:04:58.9388369Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2022-03-17T19:04:58.9389571Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2022-03-17T19:04:58.9391080Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2022-03-17T19:04:58.9392830Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2022-03-17T19:04:58.9394248Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2022-03-17T19:04:58.9395463Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2022-03-17T19:04:58.9397349Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2022-03-17T19:04:58.9398734Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2022-03-17T19:04:58.9400236Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2022-03-17T19:04:58.9401970Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2022-03-17T19:04:58.9404035Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2022-03-17T19:04:58.9405211Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2022-03-17T19:04:58.9407070Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2022-03-17T19:04:58.9408776Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2022-03-17T19:04:58.9409698Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2022-03-17T19:04:58.9411547Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2022-03-17T19:04:58.9412944Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2022-03-17T19:04:58.9414098Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2022-03-17T19:04:58.9416041Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2022-03-17T19:04:58.9417449Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2022-03-17T19:04:58.9418657Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2022-03-17T19:04:58.9420471Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2022-03-17T19:04:58.9421882Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2022-03-17T19:04:58.9423232Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2022-03-17T19:04:58.9426651Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2022-03-17T19:04:58.9427236Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2022-03-17T19:04:58.9427944Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2022-03-17T19:04:58.9430815Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2022-03-17T19:04:58.9431642Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2022-03-17T19:04:58.9432699Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2022-03-17T19:04:58.9434183Z * [new branch] gh/varal7/24/base -> origin/gh/varal7/24/base 2022-03-17T19:04:58.9435481Z * [new branch] gh/varal7/24/orig -> origin/gh/varal7/24/orig 2022-03-17T19:04:58.9437024Z * [new branch] gh/varal7/25/base -> origin/gh/varal7/25/base 2022-03-17T19:04:58.9438384Z * [new branch] gh/varal7/25/orig -> origin/gh/varal7/25/orig 2022-03-17T19:04:58.9439880Z * [new branch] gh/varal7/26/base -> origin/gh/varal7/26/base 2022-03-17T19:04:58.9441184Z * [new branch] gh/varal7/26/orig -> origin/gh/varal7/26/orig 2022-03-17T19:04:58.9443584Z * [new branch] gh/varal7/31/base -> origin/gh/varal7/31/base 2022-03-17T19:04:58.9444485Z * [new branch] gh/varal7/31/orig -> origin/gh/varal7/31/orig 2022-03-17T19:04:58.9446528Z * [new branch] gh/vec4f/1/base -> origin/gh/vec4f/1/base 2022-03-17T19:04:58.9447821Z * [new branch] gh/vec4f/1/head -> origin/gh/vec4f/1/head 2022-03-17T19:04:58.9449901Z * [new branch] gh/vincentqb/16/base -> origin/gh/vincentqb/16/base 2022-03-17T19:04:58.9451155Z * [new branch] gh/vincentqb/16/head -> origin/gh/vincentqb/16/head 2022-03-17T19:04:58.9452924Z * [new branch] gh/vincentqb/21/base -> origin/gh/vincentqb/21/base 2022-03-17T19:04:58.9454270Z * [new branch] gh/vincentqb/21/orig -> origin/gh/vincentqb/21/orig 2022-03-17T19:04:58.9456117Z * [new branch] gh/vincentqb/24/base -> origin/gh/vincentqb/24/base 2022-03-17T19:04:58.9457381Z * [new branch] gh/vincentqb/24/orig -> origin/gh/vincentqb/24/orig 2022-03-17T19:04:58.9459159Z * [new branch] gh/vincentqb/30/base -> origin/gh/vincentqb/30/base 2022-03-17T19:04:58.9460422Z * [new branch] gh/vincentqb/30/orig -> origin/gh/vincentqb/30/orig 2022-03-17T19:04:58.9462167Z * [new branch] gh/vincentqb/31/base -> origin/gh/vincentqb/31/base 2022-03-17T19:04:58.9463638Z * [new branch] gh/vincentqb/31/head -> origin/gh/vincentqb/31/head 2022-03-17T19:04:58.9464822Z * [new branch] gh/vincentqb/31/orig -> origin/gh/vincentqb/31/orig 2022-03-17T19:04:58.9466788Z * [new branch] gh/vivekmig/1/base -> origin/gh/vivekmig/1/base 2022-03-17T19:04:58.9468082Z * [new branch] gh/vivekmig/1/head -> origin/gh/vivekmig/1/head 2022-03-17T19:04:58.9469369Z * [new branch] gh/vivekmig/1/orig -> origin/gh/vivekmig/1/orig 2022-03-17T19:04:58.9471184Z * [new branch] gh/vkuzo/10/orig -> origin/gh/vkuzo/10/orig 2022-03-17T19:04:58.9472967Z * [new branch] gh/vkuzo/15/base -> origin/gh/vkuzo/15/base 2022-03-17T19:04:58.9474381Z * [new branch] gh/vkuzo/15/head -> origin/gh/vkuzo/15/head 2022-03-17T19:04:58.9475781Z * [new branch] gh/vkuzo/15/orig -> origin/gh/vkuzo/15/orig 2022-03-17T19:04:58.9477790Z * [new branch] gh/vkuzo/23/base -> origin/gh/vkuzo/23/base 2022-03-17T19:04:58.9478973Z * [new branch] gh/vkuzo/23/orig -> origin/gh/vkuzo/23/orig 2022-03-17T19:04:58.9480683Z * [new branch] gh/vkuzo/28/base -> origin/gh/vkuzo/28/base 2022-03-17T19:04:58.9482148Z * [new branch] gh/vkuzo/28/orig -> origin/gh/vkuzo/28/orig 2022-03-17T19:04:58.9484019Z * [new branch] gh/vkuzo/3/base -> origin/gh/vkuzo/3/base 2022-03-17T19:04:58.9485154Z * [new branch] gh/vkuzo/3/orig -> origin/gh/vkuzo/3/orig 2022-03-17T19:04:58.9487120Z * [new branch] gh/vkuzo/349/base -> origin/gh/vkuzo/349/base 2022-03-17T19:04:58.9488665Z * [new branch] gh/vkuzo/349/head -> origin/gh/vkuzo/349/head 2022-03-17T19:04:58.9490107Z * [new branch] gh/vkuzo/349/orig -> origin/gh/vkuzo/349/orig 2022-03-17T19:04:58.9491888Z * [new branch] gh/vkuzo/393/base -> origin/gh/vkuzo/393/base 2022-03-17T19:04:58.9493345Z * [new branch] gh/vkuzo/393/head -> origin/gh/vkuzo/393/head 2022-03-17T19:04:58.9494716Z * [new branch] gh/vkuzo/393/orig -> origin/gh/vkuzo/393/orig 2022-03-17T19:04:58.9496426Z * [new branch] gh/vkuzo/4/base -> origin/gh/vkuzo/4/base 2022-03-17T19:04:58.9497767Z * [new branch] gh/vkuzo/4/orig -> origin/gh/vkuzo/4/orig 2022-03-17T19:04:58.9499490Z * [new branch] gh/vkuzo/439/base -> origin/gh/vkuzo/439/base 2022-03-17T19:04:58.9501041Z * [new branch] gh/vkuzo/439/head -> origin/gh/vkuzo/439/head 2022-03-17T19:04:58.9502606Z * [new branch] gh/vkuzo/439/orig -> origin/gh/vkuzo/439/orig 2022-03-17T19:04:58.9504108Z * [new branch] gh/vkuzo/474/base -> origin/gh/vkuzo/474/base 2022-03-17T19:04:58.9505530Z * [new branch] gh/vkuzo/474/head -> origin/gh/vkuzo/474/head 2022-03-17T19:04:58.9506942Z * [new branch] gh/vkuzo/474/orig -> origin/gh/vkuzo/474/orig 2022-03-17T19:04:58.9508802Z * [new branch] gh/vkuzo/56/base -> origin/gh/vkuzo/56/base 2022-03-17T19:04:58.9510188Z * [new branch] gh/vkuzo/56/head -> origin/gh/vkuzo/56/head 2022-03-17T19:04:58.9511682Z * [new branch] gh/vkuzo/56/orig -> origin/gh/vkuzo/56/orig 2022-03-17T19:04:58.9513333Z * [new branch] gh/vkuzo/8/base -> origin/gh/vkuzo/8/base 2022-03-17T19:04:58.9514672Z * [new branch] gh/vkuzo/8/orig -> origin/gh/vkuzo/8/orig 2022-03-17T19:04:58.9516314Z * [new branch] gh/vkuzo/9/orig -> origin/gh/vkuzo/9/orig 2022-03-17T19:04:58.9518432Z * [new branch] gh/wanchaol/126/base -> origin/gh/wanchaol/126/base 2022-03-17T19:04:58.9519991Z * [new branch] gh/wanchaol/126/head -> origin/gh/wanchaol/126/head 2022-03-17T19:04:58.9521101Z * [new branch] gh/wanchaol/126/orig -> origin/gh/wanchaol/126/orig 2022-03-17T19:04:58.9523063Z * [new branch] gh/wanchaol/134/base -> origin/gh/wanchaol/134/base 2022-03-17T19:04:58.9524470Z * [new branch] gh/wanchaol/134/head -> origin/gh/wanchaol/134/head 2022-03-17T19:04:58.9525689Z * [new branch] gh/wanchaol/134/orig -> origin/gh/wanchaol/134/orig 2022-03-17T19:04:58.9527478Z * [new branch] gh/wanchaol/149/base -> origin/gh/wanchaol/149/base 2022-03-17T19:04:58.9528849Z * [new branch] gh/wanchaol/149/head -> origin/gh/wanchaol/149/head 2022-03-17T19:04:58.9530296Z * [new branch] gh/wanchaol/149/orig -> origin/gh/wanchaol/149/orig 2022-03-17T19:04:58.9531911Z * [new branch] gh/wanchaol/150/base -> origin/gh/wanchaol/150/base 2022-03-17T19:04:58.9533401Z * [new branch] gh/wanchaol/150/head -> origin/gh/wanchaol/150/head 2022-03-17T19:04:58.9534741Z * [new branch] gh/wanchaol/150/orig -> origin/gh/wanchaol/150/orig 2022-03-17T19:04:58.9536557Z * [new branch] gh/wanchaol/169/base -> origin/gh/wanchaol/169/base 2022-03-17T19:04:58.9537904Z * [new branch] gh/wanchaol/169/head -> origin/gh/wanchaol/169/head 2022-03-17T19:04:58.9539616Z * [new branch] gh/wanchaol/189/base -> origin/gh/wanchaol/189/base 2022-03-17T19:04:58.9540920Z * [new branch] gh/wanchaol/189/head -> origin/gh/wanchaol/189/head 2022-03-17T19:04:58.9543675Z * [new branch] gh/wanchaol/189/orig -> origin/gh/wanchaol/189/orig 2022-03-17T19:04:58.9544425Z * [new branch] gh/wanchaol/200/base -> origin/gh/wanchaol/200/base 2022-03-17T19:04:58.9545682Z * [new branch] gh/wanchaol/200/head -> origin/gh/wanchaol/200/head 2022-03-17T19:04:58.9547005Z * [new branch] gh/wanchaol/200/orig -> origin/gh/wanchaol/200/orig 2022-03-17T19:04:58.9548934Z * [new branch] gh/wanchaol/203/base -> origin/gh/wanchaol/203/base 2022-03-17T19:04:58.9550191Z * [new branch] gh/wanchaol/203/head -> origin/gh/wanchaol/203/head 2022-03-17T19:04:58.9552339Z * [new branch] gh/wanchaol/204/base -> origin/gh/wanchaol/204/base 2022-03-17T19:04:58.9553434Z * [new branch] gh/wanchaol/204/head -> origin/gh/wanchaol/204/head 2022-03-17T19:04:58.9554676Z * [new branch] gh/wanchaol/204/orig -> origin/gh/wanchaol/204/orig 2022-03-17T19:04:58.9556569Z * [new branch] gh/wanchaol/206/base -> origin/gh/wanchaol/206/base 2022-03-17T19:04:58.9557880Z * [new branch] gh/wanchaol/206/head -> origin/gh/wanchaol/206/head 2022-03-17T19:04:58.9559272Z * [new branch] gh/wanchaol/206/orig -> origin/gh/wanchaol/206/orig 2022-03-17T19:04:58.9561133Z * [new branch] gh/wanchaol/207/base -> origin/gh/wanchaol/207/base 2022-03-17T19:04:58.9562520Z * [new branch] gh/wanchaol/207/head -> origin/gh/wanchaol/207/head 2022-03-17T19:04:58.9565087Z * [new branch] gh/wanchaol/207/orig -> origin/gh/wanchaol/207/orig 2022-03-17T19:04:58.9566594Z * [new branch] gh/wanchaol/44/base -> origin/gh/wanchaol/44/base 2022-03-17T19:04:58.9567921Z * [new branch] gh/wanchaol/44/head -> origin/gh/wanchaol/44/head 2022-03-17T19:04:58.9569773Z * [new branch] gh/wanchaol/69/base -> origin/gh/wanchaol/69/base 2022-03-17T19:04:58.9571013Z * [new branch] gh/wanchaol/69/head -> origin/gh/wanchaol/69/head 2022-03-17T19:04:58.9572456Z * [new branch] gh/wanchaol/69/orig -> origin/gh/wanchaol/69/orig 2022-03-17T19:04:58.9574212Z * [new branch] gh/wanchaol/75/base -> origin/gh/wanchaol/75/base 2022-03-17T19:04:58.9575854Z * [new branch] gh/wanchaol/75/head -> origin/gh/wanchaol/75/head 2022-03-17T19:04:58.9576983Z * [new branch] gh/wanchaol/75/orig -> origin/gh/wanchaol/75/orig 2022-03-17T19:04:58.9579160Z * [new branch] gh/wenlei.xie@gmail.com/1/base -> origin/gh/wenlei.xie@gmail.com/1/base 2022-03-17T19:04:58.9580458Z * [new branch] gh/wenlei.xie@gmail.com/1/orig -> origin/gh/wenlei.xie@gmail.com/1/orig 2022-03-17T19:04:58.9581939Z * [new branch] gh/wenlei.xie@gmail.com/10/base -> origin/gh/wenlei.xie@gmail.com/10/base 2022-03-17T19:04:58.9583342Z * [new branch] gh/wenlei.xie@gmail.com/10/orig -> origin/gh/wenlei.xie@gmail.com/10/orig 2022-03-17T19:04:58.9584948Z * [new branch] gh/wenlei.xie@gmail.com/11/base -> origin/gh/wenlei.xie@gmail.com/11/base 2022-03-17T19:04:58.9586289Z * [new branch] gh/wenlei.xie@gmail.com/11/orig -> origin/gh/wenlei.xie@gmail.com/11/orig 2022-03-17T19:04:58.9588211Z * [new branch] gh/wenlei.xie@gmail.com/12/base -> origin/gh/wenlei.xie@gmail.com/12/base 2022-03-17T19:04:58.9589430Z * [new branch] gh/wenlei.xie@gmail.com/12/orig -> origin/gh/wenlei.xie@gmail.com/12/orig 2022-03-17T19:04:58.9591377Z * [new branch] gh/wenlei.xie@gmail.com/13/base -> origin/gh/wenlei.xie@gmail.com/13/base 2022-03-17T19:04:58.9592659Z * [new branch] gh/wenlei.xie@gmail.com/13/orig -> origin/gh/wenlei.xie@gmail.com/13/orig 2022-03-17T19:04:58.9594658Z * [new branch] gh/wenlei.xie@gmail.com/14/base -> origin/gh/wenlei.xie@gmail.com/14/base 2022-03-17T19:04:58.9595870Z * [new branch] gh/wenlei.xie@gmail.com/14/orig -> origin/gh/wenlei.xie@gmail.com/14/orig 2022-03-17T19:04:58.9597715Z * [new branch] gh/wenlei.xie@gmail.com/15/base -> origin/gh/wenlei.xie@gmail.com/15/base 2022-03-17T19:04:58.9598955Z * [new branch] gh/wenlei.xie@gmail.com/15/orig -> origin/gh/wenlei.xie@gmail.com/15/orig 2022-03-17T19:04:58.9600809Z * [new branch] gh/wenlei.xie@gmail.com/16/base -> origin/gh/wenlei.xie@gmail.com/16/base 2022-03-17T19:04:58.9602267Z * [new branch] gh/wenlei.xie@gmail.com/16/orig -> origin/gh/wenlei.xie@gmail.com/16/orig 2022-03-17T19:04:58.9604189Z * [new branch] gh/wenlei.xie@gmail.com/17/base -> origin/gh/wenlei.xie@gmail.com/17/base 2022-03-17T19:04:58.9605329Z * [new branch] gh/wenlei.xie@gmail.com/17/orig -> origin/gh/wenlei.xie@gmail.com/17/orig 2022-03-17T19:04:58.9607171Z * [new branch] gh/wenlei.xie@gmail.com/18/base -> origin/gh/wenlei.xie@gmail.com/18/base 2022-03-17T19:04:58.9608380Z * [new branch] gh/wenlei.xie@gmail.com/18/orig -> origin/gh/wenlei.xie@gmail.com/18/orig 2022-03-17T19:04:58.9610212Z * [new branch] gh/wenlei.xie@gmail.com/2/base -> origin/gh/wenlei.xie@gmail.com/2/base 2022-03-17T19:04:58.9611452Z * [new branch] gh/wenlei.xie@gmail.com/2/orig -> origin/gh/wenlei.xie@gmail.com/2/orig 2022-03-17T19:04:58.9612991Z * [new branch] gh/wenlei.xie@gmail.com/3/base -> origin/gh/wenlei.xie@gmail.com/3/base 2022-03-17T19:04:58.9614320Z * [new branch] gh/wenlei.xie@gmail.com/3/orig -> origin/gh/wenlei.xie@gmail.com/3/orig 2022-03-17T19:04:58.9616169Z * [new branch] gh/wenlei.xie@gmail.com/4/base -> origin/gh/wenlei.xie@gmail.com/4/base 2022-03-17T19:04:58.9617325Z * [new branch] gh/wenlei.xie@gmail.com/4/orig -> origin/gh/wenlei.xie@gmail.com/4/orig 2022-03-17T19:04:58.9619179Z * [new branch] gh/wenlei.xie@gmail.com/5/base -> origin/gh/wenlei.xie@gmail.com/5/base 2022-03-17T19:04:58.9620480Z * [new branch] gh/wenlei.xie@gmail.com/5/orig -> origin/gh/wenlei.xie@gmail.com/5/orig 2022-03-17T19:04:58.9622135Z * [new branch] gh/wenlei.xie@gmail.com/6/base -> origin/gh/wenlei.xie@gmail.com/6/base 2022-03-17T19:04:58.9623524Z * [new branch] gh/wenlei.xie@gmail.com/6/orig -> origin/gh/wenlei.xie@gmail.com/6/orig 2022-03-17T19:04:58.9625314Z * [new branch] gh/wenlei.xie@gmail.com/7/base -> origin/gh/wenlei.xie@gmail.com/7/base 2022-03-17T19:04:58.9626548Z * [new branch] gh/wenlei.xie@gmail.com/7/head -> origin/gh/wenlei.xie@gmail.com/7/head 2022-03-17T19:04:58.9627870Z * [new branch] gh/wenlei.xie@gmail.com/7/orig -> origin/gh/wenlei.xie@gmail.com/7/orig 2022-03-17T19:04:58.9629449Z * [new branch] gh/wenlei.xie@gmail.com/8/base -> origin/gh/wenlei.xie@gmail.com/8/base 2022-03-17T19:04:58.9630816Z * [new branch] gh/wenlei.xie@gmail.com/8/head -> origin/gh/wenlei.xie@gmail.com/8/head 2022-03-17T19:04:58.9632190Z * [new branch] gh/wenlei.xie@gmail.com/8/orig -> origin/gh/wenlei.xie@gmail.com/8/orig 2022-03-17T19:04:58.9633663Z * [new branch] gh/wenlei.xie@gmail.com/9/base -> origin/gh/wenlei.xie@gmail.com/9/base 2022-03-17T19:04:58.9635072Z * [new branch] gh/wenlei.xie@gmail.com/9/head -> origin/gh/wenlei.xie@gmail.com/9/head 2022-03-17T19:04:58.9636515Z * [new branch] gh/wenlei.xie@gmail.com/9/orig -> origin/gh/wenlei.xie@gmail.com/9/orig 2022-03-17T19:04:58.9638654Z * [new branch] gh/wenleix/1/base -> origin/gh/wenleix/1/base 2022-03-17T19:04:58.9639770Z * [new branch] gh/wenleix/1/orig -> origin/gh/wenleix/1/orig 2022-03-17T19:04:58.9641958Z * [new branch] gh/wenleix/18/base -> origin/gh/wenleix/18/base 2022-03-17T19:04:58.9643865Z * [new branch] gh/wenleix/18/head -> origin/gh/wenleix/18/head 2022-03-17T19:04:58.9645124Z * [new branch] gh/wenleix/18/orig -> origin/gh/wenleix/18/orig 2022-03-17T19:04:58.9646945Z * [new branch] gh/wenleix/20/base -> origin/gh/wenleix/20/base 2022-03-17T19:04:58.9648192Z * [new branch] gh/wenleix/20/head -> origin/gh/wenleix/20/head 2022-03-17T19:04:58.9649517Z * [new branch] gh/wenleix/20/orig -> origin/gh/wenleix/20/orig 2022-03-17T19:04:58.9650803Z * [new branch] gh/wenleix/21/base -> origin/gh/wenleix/21/base 2022-03-17T19:04:58.9652215Z * [new branch] gh/wenleix/21/head -> origin/gh/wenleix/21/head 2022-03-17T19:04:58.9653536Z * [new branch] gh/wenleix/21/orig -> origin/gh/wenleix/21/orig 2022-03-17T19:04:58.9655228Z * [new branch] gh/wenleix/23/base -> origin/gh/wenleix/23/base 2022-03-17T19:04:58.9656560Z * [new branch] gh/wenleix/23/head -> origin/gh/wenleix/23/head 2022-03-17T19:04:58.9657986Z * [new branch] gh/wenleix/23/orig -> origin/gh/wenleix/23/orig 2022-03-17T19:04:58.9659915Z * [new branch] gh/wenleix/24/base -> origin/gh/wenleix/24/base 2022-03-17T19:04:58.9661391Z * [new branch] gh/wenleix/24/head -> origin/gh/wenleix/24/head 2022-03-17T19:04:58.9662892Z * [new branch] gh/wenleix/24/orig -> origin/gh/wenleix/24/orig 2022-03-17T19:04:58.9664614Z * [new branch] gh/wenleix/3/base -> origin/gh/wenleix/3/base 2022-03-17T19:04:58.9665792Z * [new branch] gh/wenleix/3/orig -> origin/gh/wenleix/3/orig 2022-03-17T19:04:58.9667592Z * [new branch] gh/wenleix/30/base -> origin/gh/wenleix/30/base 2022-03-17T19:04:58.9668968Z * [new branch] gh/wenleix/30/head -> origin/gh/wenleix/30/head 2022-03-17T19:04:58.9670264Z * [new branch] gh/wenleix/30/orig -> origin/gh/wenleix/30/orig 2022-03-17T19:04:58.9671857Z * [new branch] gh/wenleix/31/base -> origin/gh/wenleix/31/base 2022-03-17T19:04:58.9673317Z * [new branch] gh/wenleix/31/head -> origin/gh/wenleix/31/head 2022-03-17T19:04:58.9674677Z * [new branch] gh/wenleix/31/orig -> origin/gh/wenleix/31/orig 2022-03-17T19:04:58.9676562Z * [new branch] gh/wenleix/32/base -> origin/gh/wenleix/32/base 2022-03-17T19:04:58.9677976Z * [new branch] gh/wenleix/32/head -> origin/gh/wenleix/32/head 2022-03-17T19:04:58.9679344Z * [new branch] gh/wenleix/32/orig -> origin/gh/wenleix/32/orig 2022-03-17T19:04:58.9681000Z * [new branch] gh/wenleix/8/base -> origin/gh/wenleix/8/base 2022-03-17T19:04:58.9682525Z * [new branch] gh/wenleix/8/orig -> origin/gh/wenleix/8/orig 2022-03-17T19:04:58.9684413Z * [new branch] gh/xcheng16/1/base -> origin/gh/xcheng16/1/base 2022-03-17T19:04:58.9685720Z * [new branch] gh/xcheng16/1/head -> origin/gh/xcheng16/1/head 2022-03-17T19:04:58.9687678Z * [new branch] gh/xcheng16/16/base -> origin/gh/xcheng16/16/base 2022-03-17T19:04:58.9688972Z * [new branch] gh/xcheng16/16/head -> origin/gh/xcheng16/16/head 2022-03-17T19:04:58.9690310Z * [new branch] gh/xcheng16/16/orig -> origin/gh/xcheng16/16/orig 2022-03-17T19:04:58.9691810Z * [new branch] gh/xcheng16/19/base -> origin/gh/xcheng16/19/base 2022-03-17T19:04:58.9693214Z * [new branch] gh/xcheng16/19/head -> origin/gh/xcheng16/19/head 2022-03-17T19:04:58.9694613Z * [new branch] gh/xcheng16/19/orig -> origin/gh/xcheng16/19/orig 2022-03-17T19:04:58.9696269Z * [new branch] gh/xcheng16/23/base -> origin/gh/xcheng16/23/base 2022-03-17T19:04:58.9697704Z * [new branch] gh/xcheng16/23/head -> origin/gh/xcheng16/23/head 2022-03-17T19:04:58.9699200Z * [new branch] gh/xcheng16/24/base -> origin/gh/xcheng16/24/base 2022-03-17T19:04:58.9700479Z * [new branch] gh/xcheng16/24/head -> origin/gh/xcheng16/24/head 2022-03-17T19:04:58.9701966Z * [new branch] gh/xcheng16/25/base -> origin/gh/xcheng16/25/base 2022-03-17T19:04:58.9703240Z * [new branch] gh/xcheng16/25/head -> origin/gh/xcheng16/25/head 2022-03-17T19:04:58.9704748Z * [new branch] gh/xcheng16/26/base -> origin/gh/xcheng16/26/base 2022-03-17T19:04:58.9705946Z * [new branch] gh/xcheng16/26/head -> origin/gh/xcheng16/26/head 2022-03-17T19:04:58.9707691Z * [new branch] gh/xcheng16/27/base -> origin/gh/xcheng16/27/base 2022-03-17T19:04:58.9709290Z * [new branch] gh/xcheng16/27/head -> origin/gh/xcheng16/27/head 2022-03-17T19:04:58.9710708Z * [new branch] gh/xcheng16/28/base -> origin/gh/xcheng16/28/base 2022-03-17T19:04:58.9711955Z * [new branch] gh/xcheng16/28/head -> origin/gh/xcheng16/28/head 2022-03-17T19:04:58.9713465Z * [new branch] gh/xcheng16/29/base -> origin/gh/xcheng16/29/base 2022-03-17T19:04:58.9714726Z * [new branch] gh/xcheng16/29/head -> origin/gh/xcheng16/29/head 2022-03-17T19:04:58.9716229Z * [new branch] gh/xcheng16/30/base -> origin/gh/xcheng16/30/base 2022-03-17T19:04:58.9717428Z * [new branch] gh/xcheng16/30/head -> origin/gh/xcheng16/30/head 2022-03-17T19:04:58.9719260Z * [new branch] gh/xcheng16/5/base -> origin/gh/xcheng16/5/base 2022-03-17T19:04:58.9721139Z * [new branch] gh/xcheng16/5/head -> origin/gh/xcheng16/5/head 2022-03-17T19:04:58.9722385Z * [new branch] gh/xcheng16/7/head -> origin/gh/xcheng16/7/head 2022-03-17T19:04:58.9725577Z * [new branch] gh/xiaobingsuper/1/base -> origin/gh/xiaobingsuper/1/base 2022-03-17T19:04:58.9726722Z * [new branch] gh/xiaobingsuper/1/orig -> origin/gh/xiaobingsuper/1/orig 2022-03-17T19:04:58.9728537Z * [new branch] gh/xiaobingsuper/2/base -> origin/gh/xiaobingsuper/2/base 2022-03-17T19:04:58.9730015Z * [new branch] gh/xiaobingsuper/2/head -> origin/gh/xiaobingsuper/2/head 2022-03-17T19:04:58.9731586Z * [new branch] gh/xiaobingsuper/6/base -> origin/gh/xiaobingsuper/6/base 2022-03-17T19:04:58.9733261Z * [new branch] gh/xiaobingsuper/6/head -> origin/gh/xiaobingsuper/6/head 2022-03-17T19:04:58.9734312Z * [new branch] gh/xiaobingsuper/6/orig -> origin/gh/xiaobingsuper/6/orig 2022-03-17T19:04:58.9736100Z * [new branch] gh/xiaobingsuper/8/base -> origin/gh/xiaobingsuper/8/base 2022-03-17T19:04:58.9737292Z * [new branch] gh/xiaobingsuper/8/orig -> origin/gh/xiaobingsuper/8/orig 2022-03-17T19:04:58.9739308Z * [new branch] gh/xta0/114/base -> origin/gh/xta0/114/base 2022-03-17T19:04:58.9740692Z * [new branch] gh/xta0/114/head -> origin/gh/xta0/114/head 2022-03-17T19:04:58.9741833Z * [new branch] gh/xta0/114/orig -> origin/gh/xta0/114/orig 2022-03-17T19:04:58.9743790Z * [new branch] gh/xta0/117/base -> origin/gh/xta0/117/base 2022-03-17T19:04:58.9745194Z * [new branch] gh/xta0/117/head -> origin/gh/xta0/117/head 2022-03-17T19:04:58.9746632Z * [new branch] gh/xta0/118/base -> origin/gh/xta0/118/base 2022-03-17T19:04:58.9747844Z * [new branch] gh/xta0/118/head -> origin/gh/xta0/118/head 2022-03-17T19:04:58.9749689Z * [new branch] gh/xta0/129/base -> origin/gh/xta0/129/base 2022-03-17T19:04:58.9751154Z * [new branch] gh/xta0/129/head -> origin/gh/xta0/129/head 2022-03-17T19:04:58.9752207Z * [new branch] gh/xta0/129/orig -> origin/gh/xta0/129/orig 2022-03-17T19:04:58.9754049Z * [new branch] gh/xta0/138/base -> origin/gh/xta0/138/base 2022-03-17T19:04:58.9755431Z * [new branch] gh/xta0/138/head -> origin/gh/xta0/138/head 2022-03-17T19:04:58.9757075Z * [new branch] gh/xta0/142/base -> origin/gh/xta0/142/base 2022-03-17T19:04:58.9758399Z * [new branch] gh/xta0/142/head -> origin/gh/xta0/142/head 2022-03-17T19:04:58.9759751Z * [new branch] gh/xta0/142/orig -> origin/gh/xta0/142/orig 2022-03-17T19:04:58.9761457Z * [new branch] gh/xta0/144/base -> origin/gh/xta0/144/base 2022-03-17T19:04:58.9763136Z * [new branch] gh/xta0/144/head -> origin/gh/xta0/144/head 2022-03-17T19:04:58.9764537Z * [new branch] gh/xta0/144/orig -> origin/gh/xta0/144/orig 2022-03-17T19:04:58.9766226Z * [new branch] gh/xta0/145/base -> origin/gh/xta0/145/base 2022-03-17T19:04:58.9767561Z * [new branch] gh/xta0/145/head -> origin/gh/xta0/145/head 2022-03-17T19:04:58.9768868Z * [new branch] gh/xta0/145/orig -> origin/gh/xta0/145/orig 2022-03-17T19:04:58.9770551Z * [new branch] gh/xta0/2/base -> origin/gh/xta0/2/base 2022-03-17T19:04:58.9772255Z * [new branch] gh/xta0/2/head -> origin/gh/xta0/2/head 2022-03-17T19:04:58.9773766Z * [new branch] gh/xta0/2/orig -> origin/gh/xta0/2/orig 2022-03-17T19:04:58.9775510Z * [new branch] gh/xta0/29/base -> origin/gh/xta0/29/base 2022-03-17T19:04:58.9777069Z * [new branch] gh/xta0/29/head -> origin/gh/xta0/29/head 2022-03-17T19:04:58.9778384Z * [new branch] gh/xta0/29/orig -> origin/gh/xta0/29/orig 2022-03-17T19:04:58.9780311Z * [new branch] gh/xta0/3/base -> origin/gh/xta0/3/base 2022-03-17T19:04:58.9781749Z * [new branch] gh/xta0/3/head -> origin/gh/xta0/3/head 2022-03-17T19:04:58.9783190Z * [new branch] gh/xta0/3/orig -> origin/gh/xta0/3/orig 2022-03-17T19:04:58.9784735Z * [new branch] gh/xta0/4/base -> origin/gh/xta0/4/base 2022-03-17T19:04:58.9786091Z * [new branch] gh/xta0/4/head -> origin/gh/xta0/4/head 2022-03-17T19:04:58.9787495Z * [new branch] gh/xta0/4/orig -> origin/gh/xta0/4/orig 2022-03-17T19:04:58.9789395Z * [new branch] gh/xta0/47/base -> origin/gh/xta0/47/base 2022-03-17T19:04:58.9790495Z * [new branch] gh/xta0/47/head -> origin/gh/xta0/47/head 2022-03-17T19:04:58.9791973Z * [new branch] gh/xta0/47/orig -> origin/gh/xta0/47/orig 2022-03-17T19:04:58.9793680Z * [new branch] gh/xta0/48/base -> origin/gh/xta0/48/base 2022-03-17T19:04:58.9794834Z * [new branch] gh/xta0/48/head -> origin/gh/xta0/48/head 2022-03-17T19:04:58.9796237Z * [new branch] gh/xta0/48/orig -> origin/gh/xta0/48/orig 2022-03-17T19:04:58.9797968Z * [new branch] gh/xta0/61/base -> origin/gh/xta0/61/base 2022-03-17T19:04:58.9799426Z * [new branch] gh/xta0/61/head -> origin/gh/xta0/61/head 2022-03-17T19:04:58.9801089Z * [new branch] gh/xta0/72/base -> origin/gh/xta0/72/base 2022-03-17T19:04:58.9803989Z * [new branch] gh/xta0/72/head -> origin/gh/xta0/72/head 2022-03-17T19:04:58.9805087Z * [new branch] gh/xta0/72/orig -> origin/gh/xta0/72/orig 2022-03-17T19:04:58.9806615Z * [new branch] gh/xta0/89/base -> origin/gh/xta0/89/base 2022-03-17T19:04:58.9807945Z * [new branch] gh/xta0/89/head -> origin/gh/xta0/89/head 2022-03-17T19:04:58.9809821Z * [new branch] gh/xuhdev/10/base -> origin/gh/xuhdev/10/base 2022-03-17T19:04:58.9811484Z * [new branch] gh/xuhdev/14/base -> origin/gh/xuhdev/14/base 2022-03-17T19:04:58.9812789Z * [new branch] gh/xuhdev/14/head -> origin/gh/xuhdev/14/head 2022-03-17T19:04:58.9814139Z * [new branch] gh/xuhdev/14/orig -> origin/gh/xuhdev/14/orig 2022-03-17T19:04:58.9815860Z * [new branch] gh/xuhdev/15/base -> origin/gh/xuhdev/15/base 2022-03-17T19:04:58.9817556Z * [new branch] gh/xuhdev/26/base -> origin/gh/xuhdev/26/base 2022-03-17T19:04:58.9818832Z * [new branch] gh/xuhdev/26/orig -> origin/gh/xuhdev/26/orig 2022-03-17T19:04:58.9820596Z * [new branch] gh/xuhdev/28/base -> origin/gh/xuhdev/28/base 2022-03-17T19:04:58.9822012Z * [new branch] gh/xuhdev/28/head -> origin/gh/xuhdev/28/head 2022-03-17T19:04:58.9823326Z * [new branch] gh/xuhdev/28/orig -> origin/gh/xuhdev/28/orig 2022-03-17T19:04:58.9825138Z * [new branch] gh/xuhdev/32/base -> origin/gh/xuhdev/32/base 2022-03-17T19:04:58.9826412Z * [new branch] gh/xuhdev/32/orig -> origin/gh/xuhdev/32/orig 2022-03-17T19:04:58.9828009Z * [new branch] gh/xuhdev/33/base -> origin/gh/xuhdev/33/base 2022-03-17T19:04:58.9829348Z * [new branch] gh/xuhdev/33/orig -> origin/gh/xuhdev/33/orig 2022-03-17T19:04:58.9830985Z * [new branch] gh/xuhdev/34/base -> origin/gh/xuhdev/34/base 2022-03-17T19:04:58.9832355Z * [new branch] gh/xuhdev/34/orig -> origin/gh/xuhdev/34/orig 2022-03-17T19:04:58.9834006Z * [new branch] gh/xuhdev/36/base -> origin/gh/xuhdev/36/base 2022-03-17T19:04:58.9835304Z * [new branch] gh/xuhdev/36/orig -> origin/gh/xuhdev/36/orig 2022-03-17T19:04:58.9836994Z * [new branch] gh/xuhdev/38/base -> origin/gh/xuhdev/38/base 2022-03-17T19:04:58.9838274Z * [new branch] gh/xuhdev/38/orig -> origin/gh/xuhdev/38/orig 2022-03-17T19:04:58.9840061Z * [new branch] gh/xuhdev/4/base -> origin/gh/xuhdev/4/base 2022-03-17T19:04:58.9841993Z * [new branch] gh/xuhdev/40/base -> origin/gh/xuhdev/40/base 2022-03-17T19:04:58.9844248Z * [new branch] gh/xuhdev/40/orig -> origin/gh/xuhdev/40/orig 2022-03-17T19:04:58.9846016Z * [new branch] gh/xuhdev/41/orig -> origin/gh/xuhdev/41/orig 2022-03-17T19:04:58.9847898Z * [new branch] gh/xuhdev/42/base -> origin/gh/xuhdev/42/base 2022-03-17T19:04:58.9849288Z * [new branch] gh/xuhdev/42/orig -> origin/gh/xuhdev/42/orig 2022-03-17T19:04:58.9851016Z * [new branch] gh/xuhdev/50/base -> origin/gh/xuhdev/50/base 2022-03-17T19:04:58.9852347Z * [new branch] gh/xuhdev/50/head -> origin/gh/xuhdev/50/head 2022-03-17T19:04:58.9853646Z * [new branch] gh/xuhdev/50/orig -> origin/gh/xuhdev/50/orig 2022-03-17T19:04:58.9855407Z * [new branch] gh/xuhdev/52/base -> origin/gh/xuhdev/52/base 2022-03-17T19:04:58.9856749Z * [new branch] gh/xuhdev/52/head -> origin/gh/xuhdev/52/head 2022-03-17T19:04:58.9858049Z * [new branch] gh/xuhdev/52/orig -> origin/gh/xuhdev/52/orig 2022-03-17T19:04:58.9859780Z * [new branch] gh/xuhdev/53/base -> origin/gh/xuhdev/53/base 2022-03-17T19:04:58.9861670Z * [new branch] gh/xuhdev/55/base -> origin/gh/xuhdev/55/base 2022-03-17T19:04:58.9863007Z * [new branch] gh/xuhdev/55/orig -> origin/gh/xuhdev/55/orig 2022-03-17T19:04:58.9864759Z * [new branch] gh/xuhdev/56/base -> origin/gh/xuhdev/56/base 2022-03-17T19:04:58.9866052Z * [new branch] gh/xuhdev/56/orig -> origin/gh/xuhdev/56/orig 2022-03-17T19:04:58.9867747Z * [new branch] gh/xuhdev/58/base -> origin/gh/xuhdev/58/base 2022-03-17T19:04:58.9869069Z * [new branch] gh/xuhdev/58/orig -> origin/gh/xuhdev/58/orig 2022-03-17T19:04:58.9870766Z * [new branch] gh/xuhdev/59/orig -> origin/gh/xuhdev/59/orig 2022-03-17T19:04:58.9872473Z * [new branch] gh/xuhdev/60/orig -> origin/gh/xuhdev/60/orig 2022-03-17T19:04:58.9874137Z * [new branch] gh/xuhdev/61/base -> origin/gh/xuhdev/61/base 2022-03-17T19:04:58.9875486Z * [new branch] gh/xuhdev/61/orig -> origin/gh/xuhdev/61/orig 2022-03-17T19:04:58.9877108Z * [new branch] gh/xuhdev/67/orig -> origin/gh/xuhdev/67/orig 2022-03-17T19:04:58.9878803Z * [new branch] gh/xuhdev/69/base -> origin/gh/xuhdev/69/base 2022-03-17T19:04:58.9880115Z * [new branch] gh/xuhdev/69/orig -> origin/gh/xuhdev/69/orig 2022-03-17T19:04:58.9881780Z * [new branch] gh/xuhdev/70/orig -> origin/gh/xuhdev/70/orig 2022-03-17T19:04:58.9883763Z * [new branch] gh/xuhdev/73/base -> origin/gh/xuhdev/73/base 2022-03-17T19:04:58.9885095Z * [new branch] gh/xuhdev/73/orig -> origin/gh/xuhdev/73/orig 2022-03-17T19:04:58.9887218Z * [new branch] gh/xuhdev/74/orig -> origin/gh/xuhdev/74/orig 2022-03-17T19:04:58.9888796Z * [new branch] gh/xuhdev/77/base -> origin/gh/xuhdev/77/base 2022-03-17T19:04:58.9890125Z * [new branch] gh/xuhdev/77/orig -> origin/gh/xuhdev/77/orig 2022-03-17T19:04:58.9891917Z * [new branch] gh/xuhdev/78/base -> origin/gh/xuhdev/78/base 2022-03-17T19:04:58.9893406Z * [new branch] gh/xuhdev/78/orig -> origin/gh/xuhdev/78/orig 2022-03-17T19:04:58.9895073Z * [new branch] gh/xuhdev/79/base -> origin/gh/xuhdev/79/base 2022-03-17T19:04:58.9896493Z * [new branch] gh/xuhdev/79/orig -> origin/gh/xuhdev/79/orig 2022-03-17T19:04:58.9898249Z * [new branch] gh/xuhdev/81/base -> origin/gh/xuhdev/81/base 2022-03-17T19:04:58.9899555Z * [new branch] gh/xuhdev/81/orig -> origin/gh/xuhdev/81/orig 2022-03-17T19:04:58.9901306Z * [new branch] gh/xuhdev/82/base -> origin/gh/xuhdev/82/base 2022-03-17T19:04:58.9902599Z * [new branch] gh/xuhdev/82/orig -> origin/gh/xuhdev/82/orig 2022-03-17T19:04:58.9904490Z * [new branch] gh/xuhdev/83/base -> origin/gh/xuhdev/83/base 2022-03-17T19:04:58.9905584Z * [new branch] gh/xuhdev/83/orig -> origin/gh/xuhdev/83/orig 2022-03-17T19:04:58.9907277Z * [new branch] gh/xuhdev/87/base -> origin/gh/xuhdev/87/base 2022-03-17T19:04:58.9908636Z * [new branch] gh/xuhdev/87/orig -> origin/gh/xuhdev/87/orig 2022-03-17T19:04:58.9910289Z * [new branch] gh/xuhdev/88/base -> origin/gh/xuhdev/88/base 2022-03-17T19:04:58.9911483Z * [new branch] gh/xuhdev/88/orig -> origin/gh/xuhdev/88/orig 2022-03-17T19:04:58.9913001Z * [new branch] gh/xuhdev/89/base -> origin/gh/xuhdev/89/base 2022-03-17T19:04:58.9914869Z * [new branch] gh/xuhdev/89/orig -> origin/gh/xuhdev/89/orig 2022-03-17T19:04:58.9916443Z * [new branch] gh/xuhdev/9/base -> origin/gh/xuhdev/9/base 2022-03-17T19:04:58.9917635Z * [new branch] gh/xuhdev/9/head -> origin/gh/xuhdev/9/head 2022-03-17T19:04:58.9919283Z * [new branch] gh/xuhdev/9/orig -> origin/gh/xuhdev/9/orig 2022-03-17T19:04:58.9921205Z * [new branch] gh/xuhdev/96/base -> origin/gh/xuhdev/96/base 2022-03-17T19:04:58.9922889Z * [new branch] gh/xuhdev/96/head -> origin/gh/xuhdev/96/head 2022-03-17T19:04:58.9924097Z * [new branch] gh/xuhdev/96/orig -> origin/gh/xuhdev/96/orig 2022-03-17T19:04:58.9926112Z * [new branch] gh/xush6528/18/base -> origin/gh/xush6528/18/base 2022-03-17T19:04:58.9927323Z * [new branch] gh/xush6528/18/head -> origin/gh/xush6528/18/head 2022-03-17T19:04:58.9929332Z * [new branch] gh/xush6528/54/base -> origin/gh/xush6528/54/base 2022-03-17T19:04:58.9930704Z * [new branch] gh/xush6528/54/head -> origin/gh/xush6528/54/head 2022-03-17T19:04:58.9931900Z * [new branch] gh/xush6528/54/orig -> origin/gh/xush6528/54/orig 2022-03-17T19:04:58.9933653Z * [new branch] gh/xush6528/58/base -> origin/gh/xush6528/58/base 2022-03-17T19:04:58.9935170Z * [new branch] gh/xush6528/58/head -> origin/gh/xush6528/58/head 2022-03-17T19:04:58.9936931Z * [new branch] gh/xuzhao9/1/base -> origin/gh/xuzhao9/1/base 2022-03-17T19:04:58.9938290Z * [new branch] gh/xuzhao9/1/orig -> origin/gh/xuzhao9/1/orig 2022-03-17T19:04:58.9939937Z * [new branch] gh/xuzhao9/2/base -> origin/gh/xuzhao9/2/base 2022-03-17T19:04:58.9941258Z * [new branch] gh/xuzhao9/2/orig -> origin/gh/xuzhao9/2/orig 2022-03-17T19:04:58.9942844Z * [new branch] gh/xuzhao9/3/base -> origin/gh/xuzhao9/3/base 2022-03-17T19:04:58.9944187Z * [new branch] gh/xuzhao9/3/orig -> origin/gh/xuzhao9/3/orig 2022-03-17T19:04:58.9946127Z * [new branch] gh/yf225/107/base -> origin/gh/yf225/107/base 2022-03-17T19:04:58.9947462Z * [new branch] gh/yf225/107/head -> origin/gh/yf225/107/head 2022-03-17T19:04:58.9948495Z * [new branch] gh/yf225/107/orig -> origin/gh/yf225/107/orig 2022-03-17T19:04:58.9950261Z * [new branch] gh/yf225/14/head -> origin/gh/yf225/14/head 2022-03-17T19:04:58.9951919Z * [new branch] gh/yf225/15/base -> origin/gh/yf225/15/base 2022-03-17T19:04:58.9953261Z * [new branch] gh/yf225/15/head -> origin/gh/yf225/15/head 2022-03-17T19:04:58.9954895Z * [new branch] gh/yf225/73/base -> origin/gh/yf225/73/base 2022-03-17T19:04:58.9956244Z * [new branch] gh/yf225/73/head -> origin/gh/yf225/73/head 2022-03-17T19:04:58.9957542Z * [new branch] gh/yf225/73/orig -> origin/gh/yf225/73/orig 2022-03-17T19:04:58.9959009Z * [new branch] gh/yf225/82/base -> origin/gh/yf225/82/base 2022-03-17T19:04:58.9960621Z * [new branch] gh/yf225/82/head -> origin/gh/yf225/82/head 2022-03-17T19:04:58.9961633Z * [new branch] gh/yf225/82/orig -> origin/gh/yf225/82/orig 2022-03-17T19:04:58.9964379Z * [new branch] gh/yf225/84/base -> origin/gh/yf225/84/base 2022-03-17T19:04:58.9965707Z * [new branch] gh/yf225/84/head -> origin/gh/yf225/84/head 2022-03-17T19:04:58.9966996Z * [new branch] gh/yf225/84/orig -> origin/gh/yf225/84/orig 2022-03-17T19:04:58.9968514Z * [new branch] gh/yf225/90/base -> origin/gh/yf225/90/base 2022-03-17T19:04:58.9969807Z * [new branch] gh/yf225/90/head -> origin/gh/yf225/90/head 2022-03-17T19:04:58.9971277Z * [new branch] gh/yf225/90/orig -> origin/gh/yf225/90/orig 2022-03-17T19:04:58.9972628Z * [new branch] gh/yf225/91/base -> origin/gh/yf225/91/base 2022-03-17T19:04:58.9973966Z * [new branch] gh/yf225/91/head -> origin/gh/yf225/91/head 2022-03-17T19:04:58.9975116Z * [new branch] gh/yf225/91/orig -> origin/gh/yf225/91/orig 2022-03-17T19:04:58.9977240Z * [new branch] gh/ysiraichi/31/base -> origin/gh/ysiraichi/31/base 2022-03-17T19:04:58.9978322Z * [new branch] gh/ysiraichi/31/head -> origin/gh/ysiraichi/31/head 2022-03-17T19:04:58.9979832Z * [new branch] gh/ysiraichi/31/orig -> origin/gh/ysiraichi/31/orig 2022-03-17T19:04:58.9981613Z * [new branch] gh/ysiraichi/35/base -> origin/gh/ysiraichi/35/base 2022-03-17T19:04:58.9983026Z * [new branch] gh/ysiraichi/35/head -> origin/gh/ysiraichi/35/head 2022-03-17T19:04:58.9984431Z * [new branch] gh/ysiraichi/35/orig -> origin/gh/ysiraichi/35/orig 2022-03-17T19:04:58.9986155Z * [new branch] gh/ysiraichi/36/base -> origin/gh/ysiraichi/36/base 2022-03-17T19:04:58.9987507Z * [new branch] gh/ysiraichi/36/orig -> origin/gh/ysiraichi/36/orig 2022-03-17T19:04:58.9989211Z * [new branch] gh/ysiraichi/37/base -> origin/gh/ysiraichi/37/base 2022-03-17T19:04:58.9990597Z * [new branch] gh/ysiraichi/37/head -> origin/gh/ysiraichi/37/head 2022-03-17T19:04:58.9992017Z * [new branch] gh/ysiraichi/37/orig -> origin/gh/ysiraichi/37/orig 2022-03-17T19:04:58.9993943Z * [new branch] gh/ysiraichi/38/base -> origin/gh/ysiraichi/38/base 2022-03-17T19:04:58.9995539Z * [new branch] gh/ysiraichi/38/head -> origin/gh/ysiraichi/38/head 2022-03-17T19:04:58.9996913Z * [new branch] gh/ysiraichi/38/orig -> origin/gh/ysiraichi/38/orig 2022-03-17T19:04:58.9998469Z * [new branch] gh/ysiraichi/39/base -> origin/gh/ysiraichi/39/base 2022-03-17T19:04:58.9999940Z * [new branch] gh/ysiraichi/39/head -> origin/gh/ysiraichi/39/head 2022-03-17T19:04:59.0000989Z * [new branch] gh/ysiraichi/39/orig -> origin/gh/ysiraichi/39/orig 2022-03-17T19:04:59.0003088Z * [new branch] gh/ysiraichi/40/base -> origin/gh/ysiraichi/40/base 2022-03-17T19:04:59.0004506Z * [new branch] gh/ysiraichi/40/head -> origin/gh/ysiraichi/40/head 2022-03-17T19:04:59.0005906Z * [new branch] gh/ysiraichi/40/orig -> origin/gh/ysiraichi/40/orig 2022-03-17T19:04:59.0007440Z * [new branch] gh/ysiraichi/41/base -> origin/gh/ysiraichi/41/base 2022-03-17T19:04:59.0008560Z * [new branch] gh/ysiraichi/41/head -> origin/gh/ysiraichi/41/head 2022-03-17T19:04:59.0010039Z * [new branch] gh/ysiraichi/41/orig -> origin/gh/ysiraichi/41/orig 2022-03-17T19:04:59.0012274Z * [new branch] gh/ysiraichi/42/base -> origin/gh/ysiraichi/42/base 2022-03-17T19:04:59.0013430Z * [new branch] gh/ysiraichi/42/head -> origin/gh/ysiraichi/42/head 2022-03-17T19:04:59.0015072Z * [new branch] gh/ysiraichi/42/orig -> origin/gh/ysiraichi/42/orig 2022-03-17T19:04:59.0016832Z * [new branch] gh/ysiraichi/43/base -> origin/gh/ysiraichi/43/base 2022-03-17T19:04:59.0018222Z * [new branch] gh/ysiraichi/43/head -> origin/gh/ysiraichi/43/head 2022-03-17T19:04:59.0019604Z * [new branch] gh/ysiraichi/43/orig -> origin/gh/ysiraichi/43/orig 2022-03-17T19:04:59.0021378Z * [new branch] gh/ysiraichi/44/base -> origin/gh/ysiraichi/44/base 2022-03-17T19:04:59.0022971Z * [new branch] gh/ysiraichi/44/head -> origin/gh/ysiraichi/44/head 2022-03-17T19:04:59.0024324Z * [new branch] gh/ysiraichi/44/orig -> origin/gh/ysiraichi/44/orig 2022-03-17T19:04:59.0026335Z * [new branch] gh/z-a-f/100/base -> origin/gh/z-a-f/100/base 2022-03-17T19:04:59.0027729Z * [new branch] gh/z-a-f/100/head -> origin/gh/z-a-f/100/head 2022-03-17T19:04:59.0029074Z * [new branch] gh/z-a-f/100/orig -> origin/gh/z-a-f/100/orig 2022-03-17T19:04:59.0030620Z * [new branch] gh/z-a-f/101/base -> origin/gh/z-a-f/101/base 2022-03-17T19:04:59.0032074Z * [new branch] gh/z-a-f/101/head -> origin/gh/z-a-f/101/head 2022-03-17T19:04:59.0033160Z * [new branch] gh/z-a-f/101/orig -> origin/gh/z-a-f/101/orig 2022-03-17T19:04:59.0034961Z * [new branch] gh/z-a-f/102/base -> origin/gh/z-a-f/102/base 2022-03-17T19:04:59.0036316Z * [new branch] gh/z-a-f/102/head -> origin/gh/z-a-f/102/head 2022-03-17T19:04:59.0037420Z * [new branch] gh/z-a-f/102/orig -> origin/gh/z-a-f/102/orig 2022-03-17T19:04:59.0039185Z * [new branch] gh/z-a-f/106/base -> origin/gh/z-a-f/106/base 2022-03-17T19:04:59.0040587Z * [new branch] gh/z-a-f/106/head -> origin/gh/z-a-f/106/head 2022-03-17T19:04:59.0041673Z * [new branch] gh/z-a-f/106/orig -> origin/gh/z-a-f/106/orig 2022-03-17T19:04:59.0044053Z * [new branch] gh/z-a-f/109/base -> origin/gh/z-a-f/109/base 2022-03-17T19:04:59.0045105Z * [new branch] gh/z-a-f/109/head -> origin/gh/z-a-f/109/head 2022-03-17T19:04:59.0046539Z * [new branch] gh/z-a-f/109/orig -> origin/gh/z-a-f/109/orig 2022-03-17T19:04:59.0048363Z * [new branch] gh/z-a-f/112/base -> origin/gh/z-a-f/112/base 2022-03-17T19:04:59.0049699Z * [new branch] gh/z-a-f/112/head -> origin/gh/z-a-f/112/head 2022-03-17T19:04:59.0051031Z * [new branch] gh/z-a-f/112/orig -> origin/gh/z-a-f/112/orig 2022-03-17T19:04:59.0052752Z * [new branch] gh/z-a-f/113/base -> origin/gh/z-a-f/113/base 2022-03-17T19:04:59.0054338Z * [new branch] gh/z-a-f/113/head -> origin/gh/z-a-f/113/head 2022-03-17T19:04:59.0056038Z * [new branch] gh/z-a-f/121/base -> origin/gh/z-a-f/121/base 2022-03-17T19:04:59.0057385Z * [new branch] gh/z-a-f/121/head -> origin/gh/z-a-f/121/head 2022-03-17T19:04:59.0058543Z * [new branch] gh/z-a-f/121/orig -> origin/gh/z-a-f/121/orig 2022-03-17T19:04:59.0060461Z * [new branch] gh/z-a-f/122/base -> origin/gh/z-a-f/122/base 2022-03-17T19:04:59.0061884Z * [new branch] gh/z-a-f/122/head -> origin/gh/z-a-f/122/head 2022-03-17T19:04:59.0062970Z * [new branch] gh/z-a-f/122/orig -> origin/gh/z-a-f/122/orig 2022-03-17T19:04:59.0064860Z * [new branch] gh/z-a-f/123/base -> origin/gh/z-a-f/123/base 2022-03-17T19:04:59.0066001Z * [new branch] gh/z-a-f/123/head -> origin/gh/z-a-f/123/head 2022-03-17T19:04:59.0067686Z * [new branch] gh/z-a-f/123/orig -> origin/gh/z-a-f/123/orig 2022-03-17T19:04:59.0069465Z * [new branch] gh/z-a-f/124/base -> origin/gh/z-a-f/124/base 2022-03-17T19:04:59.0070862Z * [new branch] gh/z-a-f/124/head -> origin/gh/z-a-f/124/head 2022-03-17T19:04:59.0072186Z * [new branch] gh/z-a-f/124/orig -> origin/gh/z-a-f/124/orig 2022-03-17T19:04:59.0073961Z * [new branch] gh/z-a-f/127/base -> origin/gh/z-a-f/127/base 2022-03-17T19:04:59.0075303Z * [new branch] gh/z-a-f/127/head -> origin/gh/z-a-f/127/head 2022-03-17T19:04:59.0076586Z * [new branch] gh/z-a-f/127/orig -> origin/gh/z-a-f/127/orig 2022-03-17T19:04:59.0078421Z * [new branch] gh/z-a-f/129/base -> origin/gh/z-a-f/129/base 2022-03-17T19:04:59.0079846Z * [new branch] gh/z-a-f/129/head -> origin/gh/z-a-f/129/head 2022-03-17T19:04:59.0081161Z * [new branch] gh/z-a-f/129/orig -> origin/gh/z-a-f/129/orig 2022-03-17T19:04:59.0083276Z * [new branch] gh/z-a-f/130/base -> origin/gh/z-a-f/130/base 2022-03-17T19:04:59.0084635Z * [new branch] gh/z-a-f/130/head -> origin/gh/z-a-f/130/head 2022-03-17T19:04:59.0086083Z * [new branch] gh/z-a-f/130/orig -> origin/gh/z-a-f/130/orig 2022-03-17T19:04:59.0087758Z * [new branch] gh/z-a-f/131/base -> origin/gh/z-a-f/131/base 2022-03-17T19:04:59.0089134Z * [new branch] gh/z-a-f/131/head -> origin/gh/z-a-f/131/head 2022-03-17T19:04:59.0090486Z * [new branch] gh/z-a-f/131/orig -> origin/gh/z-a-f/131/orig 2022-03-17T19:04:59.0092146Z * [new branch] gh/z-a-f/30/base -> origin/gh/z-a-f/30/base 2022-03-17T19:04:59.0093498Z * [new branch] gh/z-a-f/30/head -> origin/gh/z-a-f/30/head 2022-03-17T19:04:59.0094826Z * [new branch] gh/z-a-f/30/orig -> origin/gh/z-a-f/30/orig 2022-03-17T19:04:59.0096748Z * [new branch] gh/z-a-f/44/base -> origin/gh/z-a-f/44/base 2022-03-17T19:04:59.0098153Z * [new branch] gh/z-a-f/44/head -> origin/gh/z-a-f/44/head 2022-03-17T19:04:59.0099486Z * [new branch] gh/z-a-f/44/orig -> origin/gh/z-a-f/44/orig 2022-03-17T19:04:59.0101144Z * [new branch] gh/z-a-f/45/base -> origin/gh/z-a-f/45/base 2022-03-17T19:04:59.0102494Z * [new branch] gh/z-a-f/45/head -> origin/gh/z-a-f/45/head 2022-03-17T19:04:59.0103805Z * [new branch] gh/z-a-f/45/orig -> origin/gh/z-a-f/45/orig 2022-03-17T19:04:59.0105261Z * [new branch] gh/z-a-f/46/base -> origin/gh/z-a-f/46/base 2022-03-17T19:04:59.0106600Z * [new branch] gh/z-a-f/46/head -> origin/gh/z-a-f/46/head 2022-03-17T19:04:59.0107916Z * [new branch] gh/z-a-f/46/orig -> origin/gh/z-a-f/46/orig 2022-03-17T19:04:59.0109366Z * [new branch] gh/z-a-f/47/base -> origin/gh/z-a-f/47/base 2022-03-17T19:04:59.0110508Z * [new branch] gh/z-a-f/47/head -> origin/gh/z-a-f/47/head 2022-03-17T19:04:59.0111925Z * [new branch] gh/z-a-f/47/orig -> origin/gh/z-a-f/47/orig 2022-03-17T19:04:59.0113674Z * [new branch] gh/z-a-f/52/base -> origin/gh/z-a-f/52/base 2022-03-17T19:04:59.0114967Z * [new branch] gh/z-a-f/52/head -> origin/gh/z-a-f/52/head 2022-03-17T19:04:59.0116337Z * [new branch] gh/z-a-f/52/orig -> origin/gh/z-a-f/52/orig 2022-03-17T19:04:59.0117995Z * [new branch] gh/z-a-f/56/base -> origin/gh/z-a-f/56/base 2022-03-17T19:04:59.0119345Z * [new branch] gh/z-a-f/56/head -> origin/gh/z-a-f/56/head 2022-03-17T19:04:59.0120668Z * [new branch] gh/z-a-f/56/orig -> origin/gh/z-a-f/56/orig 2022-03-17T19:04:59.0122418Z * [new branch] gh/z-a-f/57/base -> origin/gh/z-a-f/57/base 2022-03-17T19:04:59.0124789Z * [new branch] gh/z-a-f/57/head -> origin/gh/z-a-f/57/head 2022-03-17T19:04:59.0125836Z * [new branch] gh/z-a-f/57/orig -> origin/gh/z-a-f/57/orig 2022-03-17T19:04:59.0127577Z * [new branch] gh/z-a-f/58/base -> origin/gh/z-a-f/58/base 2022-03-17T19:04:59.0129034Z * [new branch] gh/z-a-f/58/head -> origin/gh/z-a-f/58/head 2022-03-17T19:04:59.0130337Z * [new branch] gh/z-a-f/58/orig -> origin/gh/z-a-f/58/orig 2022-03-17T19:04:59.0132045Z * [new branch] gh/z-a-f/70/base -> origin/gh/z-a-f/70/base 2022-03-17T19:04:59.0133407Z * [new branch] gh/z-a-f/70/head -> origin/gh/z-a-f/70/head 2022-03-17T19:04:59.0134716Z * [new branch] gh/z-a-f/70/orig -> origin/gh/z-a-f/70/orig 2022-03-17T19:04:59.0136466Z * [new branch] gh/z-a-f/72/base -> origin/gh/z-a-f/72/base 2022-03-17T19:04:59.0137885Z * [new branch] gh/z-a-f/72/head -> origin/gh/z-a-f/72/head 2022-03-17T19:04:59.0139450Z * [new branch] gh/z-a-f/72/orig -> origin/gh/z-a-f/72/orig 2022-03-17T19:04:59.0140934Z * [new branch] gh/z-a-f/73/base -> origin/gh/z-a-f/73/base 2022-03-17T19:04:59.0142298Z * [new branch] gh/z-a-f/73/head -> origin/gh/z-a-f/73/head 2022-03-17T19:04:59.0143635Z * [new branch] gh/z-a-f/73/orig -> origin/gh/z-a-f/73/orig 2022-03-17T19:04:59.0145625Z * [new branch] gh/z-a-f/74/base -> origin/gh/z-a-f/74/base 2022-03-17T19:04:59.0146582Z * [new branch] gh/z-a-f/74/head -> origin/gh/z-a-f/74/head 2022-03-17T19:04:59.0148026Z * [new branch] gh/z-a-f/74/orig -> origin/gh/z-a-f/74/orig 2022-03-17T19:04:59.0149538Z * [new branch] gh/z-a-f/76/base -> origin/gh/z-a-f/76/base 2022-03-17T19:04:59.0150858Z * [new branch] gh/z-a-f/76/head -> origin/gh/z-a-f/76/head 2022-03-17T19:04:59.0152201Z * [new branch] gh/z-a-f/76/orig -> origin/gh/z-a-f/76/orig 2022-03-17T19:04:59.0154074Z * [new branch] gh/z-a-f/77/base -> origin/gh/z-a-f/77/base 2022-03-17T19:04:59.0155314Z * [new branch] gh/z-a-f/77/head -> origin/gh/z-a-f/77/head 2022-03-17T19:04:59.0156632Z * [new branch] gh/z-a-f/77/orig -> origin/gh/z-a-f/77/orig 2022-03-17T19:04:59.0158111Z * [new branch] gh/z-a-f/78/base -> origin/gh/z-a-f/78/base 2022-03-17T19:04:59.0159584Z * [new branch] gh/z-a-f/78/head -> origin/gh/z-a-f/78/head 2022-03-17T19:04:59.0160866Z * [new branch] gh/z-a-f/78/orig -> origin/gh/z-a-f/78/orig 2022-03-17T19:04:59.0162680Z * [new branch] gh/z-a-f/80/base -> origin/gh/z-a-f/80/base 2022-03-17T19:04:59.0164360Z * [new branch] gh/z-a-f/80/head -> origin/gh/z-a-f/80/head 2022-03-17T19:04:59.0165335Z * [new branch] gh/z-a-f/80/orig -> origin/gh/z-a-f/80/orig 2022-03-17T19:04:59.0167473Z * [new branch] gh/z-a-f/81/base -> origin/gh/z-a-f/81/base 2022-03-17T19:04:59.0168598Z * [new branch] gh/z-a-f/81/head -> origin/gh/z-a-f/81/head 2022-03-17T19:04:59.0170134Z * [new branch] gh/z-a-f/81/orig -> origin/gh/z-a-f/81/orig 2022-03-17T19:04:59.0171864Z * [new branch] gh/z-a-f/82/base -> origin/gh/z-a-f/82/base 2022-03-17T19:04:59.0173003Z * [new branch] gh/z-a-f/82/head -> origin/gh/z-a-f/82/head 2022-03-17T19:04:59.0174535Z * [new branch] gh/z-a-f/82/orig -> origin/gh/z-a-f/82/orig 2022-03-17T19:04:59.0176032Z * [new branch] gh/z-a-f/83/base -> origin/gh/z-a-f/83/base 2022-03-17T19:04:59.0177372Z * [new branch] gh/z-a-f/83/head -> origin/gh/z-a-f/83/head 2022-03-17T19:04:59.0178906Z * [new branch] gh/z-a-f/83/orig -> origin/gh/z-a-f/83/orig 2022-03-17T19:04:59.0180362Z * [new branch] gh/z-a-f/84/base -> origin/gh/z-a-f/84/base 2022-03-17T19:04:59.0181471Z * [new branch] gh/z-a-f/84/head -> origin/gh/z-a-f/84/head 2022-03-17T19:04:59.0182882Z * [new branch] gh/z-a-f/84/orig -> origin/gh/z-a-f/84/orig 2022-03-17T19:04:59.0184549Z * [new branch] gh/z-a-f/87/base -> origin/gh/z-a-f/87/base 2022-03-17T19:04:59.0185951Z * [new branch] gh/z-a-f/87/head -> origin/gh/z-a-f/87/head 2022-03-17T19:04:59.0187272Z * [new branch] gh/z-a-f/87/orig -> origin/gh/z-a-f/87/orig 2022-03-17T19:04:59.0188767Z * [new branch] gh/z-a-f/88/base -> origin/gh/z-a-f/88/base 2022-03-17T19:04:59.0189897Z * [new branch] gh/z-a-f/88/head -> origin/gh/z-a-f/88/head 2022-03-17T19:04:59.0191388Z * [new branch] gh/z-a-f/88/orig -> origin/gh/z-a-f/88/orig 2022-03-17T19:04:59.0193001Z * [new branch] gh/z-a-f/89/base -> origin/gh/z-a-f/89/base 2022-03-17T19:04:59.0194143Z * [new branch] gh/z-a-f/89/head -> origin/gh/z-a-f/89/head 2022-03-17T19:04:59.0195583Z * [new branch] gh/z-a-f/89/orig -> origin/gh/z-a-f/89/orig 2022-03-17T19:04:59.0197610Z * [new branch] gh/zafartahirov/1/base -> origin/gh/zafartahirov/1/base 2022-03-17T19:04:59.0199420Z * [new branch] gh/zafartahirov/101/base -> origin/gh/zafartahirov/101/base 2022-03-17T19:04:59.0200912Z * [new branch] gh/zafartahirov/101/head -> origin/gh/zafartahirov/101/head 2022-03-17T19:04:59.0202494Z * [new branch] gh/zafartahirov/101/orig -> origin/gh/zafartahirov/101/orig 2022-03-17T19:04:59.0204409Z * [new branch] gh/zafartahirov/104/base -> origin/gh/zafartahirov/104/base 2022-03-17T19:04:59.0205548Z * [new branch] gh/zafartahirov/104/orig -> origin/gh/zafartahirov/104/orig 2022-03-17T19:04:59.0207479Z * [new branch] gh/zafartahirov/106/base -> origin/gh/zafartahirov/106/base 2022-03-17T19:04:59.0208701Z * [new branch] gh/zafartahirov/106/head -> origin/gh/zafartahirov/106/head 2022-03-17T19:04:59.0210107Z * [new branch] gh/zafartahirov/106/orig -> origin/gh/zafartahirov/106/orig 2022-03-17T19:04:59.0211779Z * [new branch] gh/zafartahirov/114/base -> origin/gh/zafartahirov/114/base 2022-03-17T19:04:59.0213134Z * [new branch] gh/zafartahirov/114/head -> origin/gh/zafartahirov/114/head 2022-03-17T19:04:59.0214631Z * [new branch] gh/zafartahirov/115/base -> origin/gh/zafartahirov/115/base 2022-03-17T19:04:59.0216060Z * [new branch] gh/zafartahirov/115/head -> origin/gh/zafartahirov/115/head 2022-03-17T19:04:59.0217539Z * [new branch] gh/zafartahirov/116/base -> origin/gh/zafartahirov/116/base 2022-03-17T19:04:59.0218990Z * [new branch] gh/zafartahirov/116/head -> origin/gh/zafartahirov/116/head 2022-03-17T19:04:59.0220798Z * [new branch] gh/zafartahirov/118/base -> origin/gh/zafartahirov/118/base 2022-03-17T19:04:59.0222447Z * [new branch] gh/zafartahirov/118/head -> origin/gh/zafartahirov/118/head 2022-03-17T19:04:59.0223636Z * [new branch] gh/zafartahirov/118/orig -> origin/gh/zafartahirov/118/orig 2022-03-17T19:04:59.0225563Z * [new branch] gh/zafartahirov/119/base -> origin/gh/zafartahirov/119/base 2022-03-17T19:04:59.0226716Z * [new branch] gh/zafartahirov/119/head -> origin/gh/zafartahirov/119/head 2022-03-17T19:04:59.0228187Z * [new branch] gh/zafartahirov/119/orig -> origin/gh/zafartahirov/119/orig 2022-03-17T19:04:59.0229709Z * [new branch] gh/zafartahirov/120/base -> origin/gh/zafartahirov/120/base 2022-03-17T19:04:59.0231223Z * [new branch] gh/zafartahirov/120/head -> origin/gh/zafartahirov/120/head 2022-03-17T19:04:59.0232383Z * [new branch] gh/zafartahirov/120/orig -> origin/gh/zafartahirov/120/orig 2022-03-17T19:04:59.0233968Z * [new branch] gh/zafartahirov/121/base -> origin/gh/zafartahirov/121/base 2022-03-17T19:04:59.0235309Z * [new branch] gh/zafartahirov/121/head -> origin/gh/zafartahirov/121/head 2022-03-17T19:04:59.0236648Z * [new branch] gh/zafartahirov/121/orig -> origin/gh/zafartahirov/121/orig 2022-03-17T19:04:59.0238202Z * [new branch] gh/zafartahirov/122/base -> origin/gh/zafartahirov/122/base 2022-03-17T19:04:59.0239572Z * [new branch] gh/zafartahirov/122/head -> origin/gh/zafartahirov/122/head 2022-03-17T19:04:59.0240773Z * [new branch] gh/zafartahirov/122/orig -> origin/gh/zafartahirov/122/orig 2022-03-17T19:04:59.0242772Z * [new branch] gh/zafartahirov/123/base -> origin/gh/zafartahirov/123/base 2022-03-17T19:04:59.0244158Z * [new branch] gh/zafartahirov/123/head -> origin/gh/zafartahirov/123/head 2022-03-17T19:04:59.0246040Z * [new branch] gh/zafartahirov/124/base -> origin/gh/zafartahirov/124/base 2022-03-17T19:04:59.0247242Z * [new branch] gh/zafartahirov/124/head -> origin/gh/zafartahirov/124/head 2022-03-17T19:04:59.0248816Z * [new branch] gh/zafartahirov/124/orig -> origin/gh/zafartahirov/124/orig 2022-03-17T19:04:59.0250593Z * [new branch] gh/zafartahirov/125/base -> origin/gh/zafartahirov/125/base 2022-03-17T19:04:59.0251735Z * [new branch] gh/zafartahirov/125/head -> origin/gh/zafartahirov/125/head 2022-03-17T19:04:59.0253271Z * [new branch] gh/zafartahirov/125/orig -> origin/gh/zafartahirov/125/orig 2022-03-17T19:04:59.0255240Z * [new branch] gh/zafartahirov/2/base -> origin/gh/zafartahirov/2/base 2022-03-17T19:04:59.0256757Z * [new branch] gh/zafartahirov/61/base -> origin/gh/zafartahirov/61/base 2022-03-17T19:04:59.0257954Z * [new branch] gh/zafartahirov/61/head -> origin/gh/zafartahirov/61/head 2022-03-17T19:04:59.0259454Z * [new branch] gh/zafartahirov/61/orig -> origin/gh/zafartahirov/61/orig 2022-03-17T19:04:59.0261612Z * [new branch] gh/zasdfgbnm/10/base -> origin/gh/zasdfgbnm/10/base 2022-03-17T19:04:59.0263157Z * [new branch] gh/zasdfgbnm/10/head -> origin/gh/zasdfgbnm/10/head 2022-03-17T19:04:59.0264344Z * [new branch] gh/zasdfgbnm/10/orig -> origin/gh/zasdfgbnm/10/orig 2022-03-17T19:04:59.0266473Z * [new branch] gh/zasdfgbnm/14/base -> origin/gh/zasdfgbnm/14/base 2022-03-17T19:04:59.0267965Z * [new branch] gh/zasdfgbnm/14/orig -> origin/gh/zasdfgbnm/14/orig 2022-03-17T19:04:59.0269567Z * [new branch] gh/zasdfgbnm/15/base -> origin/gh/zasdfgbnm/15/base 2022-03-17T19:04:59.0270908Z * [new branch] gh/zasdfgbnm/15/orig -> origin/gh/zasdfgbnm/15/orig 2022-03-17T19:04:59.0272569Z * [new branch] gh/zasdfgbnm/16/base -> origin/gh/zasdfgbnm/16/base 2022-03-17T19:04:59.0273957Z * [new branch] gh/zasdfgbnm/16/orig -> origin/gh/zasdfgbnm/16/orig 2022-03-17T19:04:59.0275540Z * [new branch] gh/zasdfgbnm/17/orig -> origin/gh/zasdfgbnm/17/orig 2022-03-17T19:04:59.0277175Z * [new branch] gh/zasdfgbnm/18/base -> origin/gh/zasdfgbnm/18/base 2022-03-17T19:04:59.0278519Z * [new branch] gh/zasdfgbnm/18/orig -> origin/gh/zasdfgbnm/18/orig 2022-03-17T19:04:59.0280140Z * [new branch] gh/zasdfgbnm/19/orig -> origin/gh/zasdfgbnm/19/orig 2022-03-17T19:04:59.0281924Z * [new branch] gh/zasdfgbnm/20/base -> origin/gh/zasdfgbnm/20/base 2022-03-17T19:04:59.0284005Z * [new branch] gh/zasdfgbnm/20/orig -> origin/gh/zasdfgbnm/20/orig 2022-03-17T19:04:59.0285924Z * [new branch] gh/zasdfgbnm/21/base -> origin/gh/zasdfgbnm/21/base 2022-03-17T19:04:59.0287039Z * [new branch] gh/zasdfgbnm/21/head -> origin/gh/zasdfgbnm/21/head 2022-03-17T19:04:59.0288518Z * [new branch] gh/zasdfgbnm/21/orig -> origin/gh/zasdfgbnm/21/orig 2022-03-17T19:04:59.0289993Z * [new branch] gh/zasdfgbnm/22/base -> origin/gh/zasdfgbnm/22/base 2022-03-17T19:04:59.0291476Z * [new branch] gh/zasdfgbnm/22/head -> origin/gh/zasdfgbnm/22/head 2022-03-17T19:04:59.0292837Z * [new branch] gh/zasdfgbnm/22/orig -> origin/gh/zasdfgbnm/22/orig 2022-03-17T19:04:59.0294272Z * [new branch] gh/zasdfgbnm/23/base -> origin/gh/zasdfgbnm/23/base 2022-03-17T19:04:59.0295639Z * [new branch] gh/zasdfgbnm/23/head -> origin/gh/zasdfgbnm/23/head 2022-03-17T19:04:59.0296809Z * [new branch] gh/zasdfgbnm/23/orig -> origin/gh/zasdfgbnm/23/orig 2022-03-17T19:04:59.0298685Z * [new branch] gh/zasdfgbnm/24/base -> origin/gh/zasdfgbnm/24/base 2022-03-17T19:04:59.0299992Z * [new branch] gh/zasdfgbnm/24/orig -> origin/gh/zasdfgbnm/24/orig 2022-03-17T19:04:59.0301788Z * [new branch] gh/zasdfgbnm/25/base -> origin/gh/zasdfgbnm/25/base 2022-03-17T19:04:59.0303120Z * [new branch] gh/zasdfgbnm/25/orig -> origin/gh/zasdfgbnm/25/orig 2022-03-17T19:04:59.0304772Z * [new branch] gh/zasdfgbnm/26/orig -> origin/gh/zasdfgbnm/26/orig 2022-03-17T19:04:59.0306535Z * [new branch] gh/zasdfgbnm/27/orig -> origin/gh/zasdfgbnm/27/orig 2022-03-17T19:04:59.0308206Z * [new branch] gh/zasdfgbnm/28/base -> origin/gh/zasdfgbnm/28/base 2022-03-17T19:04:59.0309330Z * [new branch] gh/zasdfgbnm/28/orig -> origin/gh/zasdfgbnm/28/orig 2022-03-17T19:04:59.0311290Z * [new branch] gh/zasdfgbnm/29/base -> origin/gh/zasdfgbnm/29/base 2022-03-17T19:04:59.0312481Z * [new branch] gh/zasdfgbnm/29/orig -> origin/gh/zasdfgbnm/29/orig 2022-03-17T19:04:59.0314370Z * [new branch] gh/zasdfgbnm/30/base -> origin/gh/zasdfgbnm/30/base 2022-03-17T19:04:59.0315481Z * [new branch] gh/zasdfgbnm/30/orig -> origin/gh/zasdfgbnm/30/orig 2022-03-17T19:04:59.0316996Z * [new branch] gh/zasdfgbnm/31/orig -> origin/gh/zasdfgbnm/31/orig 2022-03-17T19:04:59.0318965Z * [new branch] gh/zasdfgbnm/32/base -> origin/gh/zasdfgbnm/32/base 2022-03-17T19:04:59.0320097Z * [new branch] gh/zasdfgbnm/32/orig -> origin/gh/zasdfgbnm/32/orig 2022-03-17T19:04:59.0321968Z * [new branch] gh/zasdfgbnm/33/base -> origin/gh/zasdfgbnm/33/base 2022-03-17T19:04:59.0323633Z * [new branch] gh/zasdfgbnm/33/orig -> origin/gh/zasdfgbnm/33/orig 2022-03-17T19:04:59.0325228Z * [new branch] gh/zasdfgbnm/34/orig -> origin/gh/zasdfgbnm/34/orig 2022-03-17T19:04:59.0326760Z * [new branch] gh/zasdfgbnm/35/base -> origin/gh/zasdfgbnm/35/base 2022-03-17T19:04:59.0328054Z * [new branch] gh/zasdfgbnm/35/head -> origin/gh/zasdfgbnm/35/head 2022-03-17T19:04:59.0329338Z * [new branch] gh/zasdfgbnm/35/orig -> origin/gh/zasdfgbnm/35/orig 2022-03-17T19:04:59.0331094Z * [new branch] gh/zasdfgbnm/36/base -> origin/gh/zasdfgbnm/36/base 2022-03-17T19:04:59.0332399Z * [new branch] gh/zasdfgbnm/36/orig -> origin/gh/zasdfgbnm/36/orig 2022-03-17T19:04:59.0334091Z * [new branch] gh/zasdfgbnm/37/base -> origin/gh/zasdfgbnm/37/base 2022-03-17T19:04:59.0335402Z * [new branch] gh/zasdfgbnm/37/orig -> origin/gh/zasdfgbnm/37/orig 2022-03-17T19:04:59.0337089Z * [new branch] gh/zasdfgbnm/38/base -> origin/gh/zasdfgbnm/38/base 2022-03-17T19:04:59.0338491Z * [new branch] gh/zasdfgbnm/38/orig -> origin/gh/zasdfgbnm/38/orig 2022-03-17T19:04:59.0340509Z * [new branch] gh/zasdfgbnm/39/base -> origin/gh/zasdfgbnm/39/base 2022-03-17T19:04:59.0341599Z * [new branch] gh/zasdfgbnm/39/orig -> origin/gh/zasdfgbnm/39/orig 2022-03-17T19:04:59.0343387Z * [new branch] gh/zasdfgbnm/40/base -> origin/gh/zasdfgbnm/40/base 2022-03-17T19:04:59.0344746Z * [new branch] gh/zasdfgbnm/40/orig -> origin/gh/zasdfgbnm/40/orig 2022-03-17T19:04:59.0346445Z * [new branch] gh/zasdfgbnm/41/base -> origin/gh/zasdfgbnm/41/base 2022-03-17T19:04:59.0347734Z * [new branch] gh/zasdfgbnm/41/orig -> origin/gh/zasdfgbnm/41/orig 2022-03-17T19:04:59.0349406Z * [new branch] gh/zasdfgbnm/42/base -> origin/gh/zasdfgbnm/42/base 2022-03-17T19:04:59.0350731Z * [new branch] gh/zasdfgbnm/42/orig -> origin/gh/zasdfgbnm/42/orig 2022-03-17T19:04:59.0352463Z * [new branch] gh/zasdfgbnm/43/base -> origin/gh/zasdfgbnm/43/base 2022-03-17T19:04:59.0354222Z * [new branch] gh/zasdfgbnm/44/orig -> origin/gh/zasdfgbnm/44/orig 2022-03-17T19:04:59.0355954Z * [new branch] gh/zasdfgbnm/45/base -> origin/gh/zasdfgbnm/45/base 2022-03-17T19:04:59.0357255Z * [new branch] gh/zasdfgbnm/45/orig -> origin/gh/zasdfgbnm/45/orig 2022-03-17T19:04:59.0359005Z * [new branch] gh/zasdfgbnm/47/base -> origin/gh/zasdfgbnm/47/base 2022-03-17T19:04:59.0360289Z * [new branch] gh/zasdfgbnm/47/orig -> origin/gh/zasdfgbnm/47/orig 2022-03-17T19:04:59.0362050Z * [new branch] gh/zasdfgbnm/48/base -> origin/gh/zasdfgbnm/48/base 2022-03-17T19:04:59.0363467Z * [new branch] gh/zasdfgbnm/48/orig -> origin/gh/zasdfgbnm/48/orig 2022-03-17T19:04:59.0364974Z * [new branch] gh/zasdfgbnm/49/base -> origin/gh/zasdfgbnm/49/base 2022-03-17T19:04:59.0366325Z * [new branch] gh/zasdfgbnm/49/orig -> origin/gh/zasdfgbnm/49/orig 2022-03-17T19:04:59.0368592Z * [new branch] gh/zasdfgbnm/50/base -> origin/gh/zasdfgbnm/50/base 2022-03-17T19:04:59.0369962Z * [new branch] gh/zasdfgbnm/50/orig -> origin/gh/zasdfgbnm/50/orig 2022-03-17T19:04:59.0371736Z * [new branch] gh/zasdfgbnm/51/base -> origin/gh/zasdfgbnm/51/base 2022-03-17T19:04:59.0373157Z * [new branch] gh/zasdfgbnm/51/orig -> origin/gh/zasdfgbnm/51/orig 2022-03-17T19:04:59.0374832Z * [new branch] gh/zasdfgbnm/52/orig -> origin/gh/zasdfgbnm/52/orig 2022-03-17T19:04:59.0376490Z * [new branch] gh/zasdfgbnm/53/orig -> origin/gh/zasdfgbnm/53/orig 2022-03-17T19:04:59.0378226Z * [new branch] gh/zasdfgbnm/54/orig -> origin/gh/zasdfgbnm/54/orig 2022-03-17T19:04:59.0379942Z * [new branch] gh/zasdfgbnm/55/orig -> origin/gh/zasdfgbnm/55/orig 2022-03-17T19:04:59.0381512Z * [new branch] gh/zasdfgbnm/56/orig -> origin/gh/zasdfgbnm/56/orig 2022-03-17T19:04:59.0383109Z * [new branch] gh/zasdfgbnm/57/orig -> origin/gh/zasdfgbnm/57/orig 2022-03-17T19:04:59.0384872Z * [new branch] gh/zasdfgbnm/58/orig -> origin/gh/zasdfgbnm/58/orig 2022-03-17T19:04:59.0386592Z * [new branch] gh/zasdfgbnm/59/orig -> origin/gh/zasdfgbnm/59/orig 2022-03-17T19:04:59.0388276Z * [new branch] gh/zasdfgbnm/60/orig -> origin/gh/zasdfgbnm/60/orig 2022-03-17T19:04:59.0389838Z * [new branch] gh/zasdfgbnm/61/orig -> origin/gh/zasdfgbnm/61/orig 2022-03-17T19:04:59.0391552Z * [new branch] gh/zasdfgbnm/62/base -> origin/gh/zasdfgbnm/62/base 2022-03-17T19:04:59.0392926Z * [new branch] gh/zasdfgbnm/62/orig -> origin/gh/zasdfgbnm/62/orig 2022-03-17T19:04:59.0394490Z * [new branch] gh/zasdfgbnm/63/orig -> origin/gh/zasdfgbnm/63/orig 2022-03-17T19:04:59.0396321Z * [new branch] gh/zasdfgbnm/64/orig -> origin/gh/zasdfgbnm/64/orig 2022-03-17T19:04:59.0397922Z * [new branch] gh/zasdfgbnm/65/orig -> origin/gh/zasdfgbnm/65/orig 2022-03-17T19:04:59.0399630Z * [new branch] gh/zasdfgbnm/66/base -> origin/gh/zasdfgbnm/66/base 2022-03-17T19:04:59.0403519Z * [new branch] gh/zasdfgbnm/66/orig -> origin/gh/zasdfgbnm/66/orig 2022-03-17T19:04:59.0404326Z * [new branch] gh/zasdfgbnm/67/orig -> origin/gh/zasdfgbnm/67/orig 2022-03-17T19:04:59.0405153Z * [new branch] gh/zasdfgbnm/68/base -> origin/gh/zasdfgbnm/68/base 2022-03-17T19:04:59.0406170Z * [new branch] gh/zasdfgbnm/68/orig -> origin/gh/zasdfgbnm/68/orig 2022-03-17T19:04:59.0407777Z * [new branch] gh/zasdfgbnm/69/orig -> origin/gh/zasdfgbnm/69/orig 2022-03-17T19:04:59.0409426Z * [new branch] gh/zasdfgbnm/70/orig -> origin/gh/zasdfgbnm/70/orig 2022-03-17T19:04:59.0411153Z * [new branch] gh/zasdfgbnm/71/orig -> origin/gh/zasdfgbnm/71/orig 2022-03-17T19:04:59.0412893Z * [new branch] gh/zasdfgbnm/72/base -> origin/gh/zasdfgbnm/72/base 2022-03-17T19:04:59.0414088Z * [new branch] gh/zasdfgbnm/72/orig -> origin/gh/zasdfgbnm/72/orig 2022-03-17T19:04:59.0416085Z * [new branch] gh/zasdfgbnm/73/base -> origin/gh/zasdfgbnm/73/base 2022-03-17T19:04:59.0417192Z * [new branch] gh/zasdfgbnm/73/orig -> origin/gh/zasdfgbnm/73/orig 2022-03-17T19:04:59.0418801Z * [new branch] gh/zasdfgbnm/74/base -> origin/gh/zasdfgbnm/74/base 2022-03-17T19:04:59.0420136Z * [new branch] gh/zasdfgbnm/74/orig -> origin/gh/zasdfgbnm/74/orig 2022-03-17T19:04:59.0421953Z * [new branch] gh/zasdfgbnm/76/base -> origin/gh/zasdfgbnm/76/base 2022-03-17T19:04:59.0423140Z * [new branch] gh/zasdfgbnm/76/orig -> origin/gh/zasdfgbnm/76/orig 2022-03-17T19:04:59.0425040Z * [new branch] gh/zasdfgbnm/77/base -> origin/gh/zasdfgbnm/77/base 2022-03-17T19:04:59.0426259Z * [new branch] gh/zasdfgbnm/77/orig -> origin/gh/zasdfgbnm/77/orig 2022-03-17T19:04:59.0428140Z * [new branch] gh/zasdfgbnm/78/orig -> origin/gh/zasdfgbnm/78/orig 2022-03-17T19:04:59.0429651Z * [new branch] gh/zasdfgbnm/79/orig -> origin/gh/zasdfgbnm/79/orig 2022-03-17T19:04:59.0431518Z * [new branch] gh/zasdfgbnm/8/base -> origin/gh/zasdfgbnm/8/base 2022-03-17T19:04:59.0432799Z * [new branch] gh/zasdfgbnm/8/head -> origin/gh/zasdfgbnm/8/head 2022-03-17T19:04:59.0434067Z * [new branch] gh/zasdfgbnm/8/orig -> origin/gh/zasdfgbnm/8/orig 2022-03-17T19:04:59.0435887Z * [new branch] gh/zasdfgbnm/80/base -> origin/gh/zasdfgbnm/80/base 2022-03-17T19:04:59.0437088Z * [new branch] gh/zasdfgbnm/80/orig -> origin/gh/zasdfgbnm/80/orig 2022-03-17T19:04:59.0438964Z * [new branch] gh/zasdfgbnm/81/base -> origin/gh/zasdfgbnm/81/base 2022-03-17T19:04:59.0440177Z * [new branch] gh/zasdfgbnm/81/orig -> origin/gh/zasdfgbnm/81/orig 2022-03-17T19:04:59.0442130Z * [new branch] gh/zasdfgbnm/82/orig -> origin/gh/zasdfgbnm/82/orig 2022-03-17T19:04:59.0444534Z * [new branch] gh/zasdfgbnm/83/base -> origin/gh/zasdfgbnm/83/base 2022-03-17T19:04:59.0445615Z * [new branch] gh/zasdfgbnm/83/orig -> origin/gh/zasdfgbnm/83/orig 2022-03-17T19:04:59.0447350Z * [new branch] gh/zasdfgbnm/84/orig -> origin/gh/zasdfgbnm/84/orig 2022-03-17T19:04:59.0448975Z * [new branch] gh/zasdfgbnm/85/base -> origin/gh/zasdfgbnm/85/base 2022-03-17T19:04:59.0450115Z * [new branch] gh/zasdfgbnm/85/orig -> origin/gh/zasdfgbnm/85/orig 2022-03-17T19:04:59.0451949Z * [new branch] gh/zasdfgbnm/86/base -> origin/gh/zasdfgbnm/86/base 2022-03-17T19:04:59.0454119Z * [new branch] gh/zasdfgbnm/87/base -> origin/gh/zasdfgbnm/87/base 2022-03-17T19:04:59.0455095Z * [new branch] gh/zasdfgbnm/87/orig -> origin/gh/zasdfgbnm/87/orig 2022-03-17T19:04:59.0456930Z * [new branch] gh/zasdfgbnm/88/base -> origin/gh/zasdfgbnm/88/base 2022-03-17T19:04:59.0458348Z * [new branch] gh/zasdfgbnm/88/orig -> origin/gh/zasdfgbnm/88/orig 2022-03-17T19:04:59.0460092Z * [new branch] gh/zasdfgbnm/89/base -> origin/gh/zasdfgbnm/89/base 2022-03-17T19:04:59.0461450Z * [new branch] gh/zasdfgbnm/89/orig -> origin/gh/zasdfgbnm/89/orig 2022-03-17T19:04:59.0462981Z * [new branch] gh/zasdfgbnm/9/base -> origin/gh/zasdfgbnm/9/base 2022-03-17T19:04:59.0463996Z * [new branch] gh/zasdfgbnm/9/head -> origin/gh/zasdfgbnm/9/head 2022-03-17T19:04:59.0465428Z * [new branch] gh/zasdfgbnm/9/orig -> origin/gh/zasdfgbnm/9/orig 2022-03-17T19:04:59.0467204Z * [new branch] gh/zasdfgbnm/90/base -> origin/gh/zasdfgbnm/90/base 2022-03-17T19:04:59.0468574Z * [new branch] gh/zasdfgbnm/90/orig -> origin/gh/zasdfgbnm/90/orig 2022-03-17T19:04:59.0470325Z * [new branch] gh/zasdfgbnm/92/base -> origin/gh/zasdfgbnm/92/base 2022-03-17T19:04:59.0471505Z * [new branch] gh/zasdfgbnm/92/orig -> origin/gh/zasdfgbnm/92/orig 2022-03-17T19:04:59.0473239Z * [new branch] gh/zasdfgbnm/93/base -> origin/gh/zasdfgbnm/93/base 2022-03-17T19:04:59.0474694Z * [new branch] gh/zasdfgbnm/93/orig -> origin/gh/zasdfgbnm/93/orig 2022-03-17T19:04:59.0476207Z * [new branch] gh/zasdfgbnm/94/base -> origin/gh/zasdfgbnm/94/base 2022-03-17T19:04:59.0477567Z * [new branch] gh/zasdfgbnm/94/orig -> origin/gh/zasdfgbnm/94/orig 2022-03-17T19:04:59.0479161Z * [new branch] gh/zasdfgbnm/95/base -> origin/gh/zasdfgbnm/95/base 2022-03-17T19:04:59.0480829Z * [new branch] gh/zasdfgbnm/95/orig -> origin/gh/zasdfgbnm/95/orig 2022-03-17T19:04:59.0482385Z * [new branch] gh/zasdfgbnm/96/base -> origin/gh/zasdfgbnm/96/base 2022-03-17T19:04:59.0483834Z * [new branch] gh/zasdfgbnm/96/orig -> origin/gh/zasdfgbnm/96/orig 2022-03-17T19:04:59.0486031Z * [new branch] gh/zdevito/118/base -> origin/gh/zdevito/118/base 2022-03-17T19:04:59.0487346Z * [new branch] gh/zdevito/118/head -> origin/gh/zdevito/118/head 2022-03-17T19:04:59.0488730Z * [new branch] gh/zdevito/118/orig -> origin/gh/zdevito/118/orig 2022-03-17T19:04:59.0490634Z * [new branch] gh/zdevito/129/base -> origin/gh/zdevito/129/base 2022-03-17T19:04:59.0491919Z * [new branch] gh/zdevito/129/head -> origin/gh/zdevito/129/head 2022-03-17T19:04:59.0493353Z * [new branch] gh/zdevito/129/orig -> origin/gh/zdevito/129/orig 2022-03-17T19:04:59.0494892Z * [new branch] gh/zdevito/143/base -> origin/gh/zdevito/143/base 2022-03-17T19:04:59.0496181Z * [new branch] gh/zdevito/143/head -> origin/gh/zdevito/143/head 2022-03-17T19:04:59.0497559Z * [new branch] gh/zdevito/143/orig -> origin/gh/zdevito/143/orig 2022-03-17T19:04:59.0500036Z * [new branch] gh/zdevito/145/base -> origin/gh/zdevito/145/base 2022-03-17T19:04:59.0500923Z * [new branch] gh/zdevito/145/head -> origin/gh/zdevito/145/head 2022-03-17T19:04:59.0502209Z * [new branch] gh/zdevito/145/orig -> origin/gh/zdevito/145/orig 2022-03-17T19:04:59.0504037Z * [new branch] gh/zdevito/159/base -> origin/gh/zdevito/159/base 2022-03-17T19:04:59.0505227Z * [new branch] gh/zdevito/159/head -> origin/gh/zdevito/159/head 2022-03-17T19:04:59.0506477Z * [new branch] gh/zdevito/159/orig -> origin/gh/zdevito/159/orig 2022-03-17T19:04:59.0508679Z * [new branch] gh/zdevito/160/base -> origin/gh/zdevito/160/base 2022-03-17T19:04:59.0509655Z * [new branch] gh/zdevito/160/head -> origin/gh/zdevito/160/head 2022-03-17T19:04:59.0510945Z * [new branch] gh/zdevito/160/orig -> origin/gh/zdevito/160/orig 2022-03-17T19:04:59.0512820Z * [new branch] gh/zdevito/161/base -> origin/gh/zdevito/161/base 2022-03-17T19:04:59.0513961Z * [new branch] gh/zdevito/161/head -> origin/gh/zdevito/161/head 2022-03-17T19:04:59.0515364Z * [new branch] gh/zdevito/161/orig -> origin/gh/zdevito/161/orig 2022-03-17T19:04:59.0517228Z * [new branch] gh/zdevito/170/base -> origin/gh/zdevito/170/base 2022-03-17T19:04:59.0518456Z * [new branch] gh/zdevito/170/head -> origin/gh/zdevito/170/head 2022-03-17T19:04:59.0519827Z * [new branch] gh/zdevito/170/orig -> origin/gh/zdevito/170/orig 2022-03-17T19:04:59.0521582Z * [new branch] gh/zdevito/171/base -> origin/gh/zdevito/171/base 2022-03-17T19:04:59.0523218Z * [new branch] gh/zdevito/171/head -> origin/gh/zdevito/171/head 2022-03-17T19:04:59.0524598Z * [new branch] gh/zdevito/171/orig -> origin/gh/zdevito/171/orig 2022-03-17T19:04:59.0526062Z * [new branch] gh/zdevito/172/base -> origin/gh/zdevito/172/base 2022-03-17T19:04:59.0527420Z * [new branch] gh/zdevito/172/head -> origin/gh/zdevito/172/head 2022-03-17T19:04:59.0528772Z * [new branch] gh/zdevito/172/orig -> origin/gh/zdevito/172/orig 2022-03-17T19:04:59.0530787Z * [new branch] gh/zdevito/174/base -> origin/gh/zdevito/174/base 2022-03-17T19:04:59.0531981Z * [new branch] gh/zdevito/174/head -> origin/gh/zdevito/174/head 2022-03-17T19:04:59.0533290Z * [new branch] gh/zdevito/174/orig -> origin/gh/zdevito/174/orig 2022-03-17T19:04:59.0535107Z * [new branch] gh/zdevito/181/base -> origin/gh/zdevito/181/base 2022-03-17T19:04:59.0536260Z * [new branch] gh/zdevito/181/head -> origin/gh/zdevito/181/head 2022-03-17T19:04:59.0537646Z * [new branch] gh/zdevito/181/orig -> origin/gh/zdevito/181/orig 2022-03-17T19:04:59.0539404Z * [new branch] gh/zdevito/183/base -> origin/gh/zdevito/183/base 2022-03-17T19:04:59.0540594Z * [new branch] gh/zdevito/183/orig -> origin/gh/zdevito/183/orig 2022-03-17T19:04:59.0542395Z * [new branch] gh/zdevito/184/base -> origin/gh/zdevito/184/base 2022-03-17T19:04:59.0543570Z * [new branch] gh/zdevito/184/head -> origin/gh/zdevito/184/head 2022-03-17T19:04:59.0544934Z * [new branch] gh/zdevito/184/orig -> origin/gh/zdevito/184/orig 2022-03-17T19:04:59.0546780Z * [new branch] gh/zdevito/99/base -> origin/gh/zdevito/99/base 2022-03-17T19:04:59.0547996Z * [new branch] gh/zdevito/99/head -> origin/gh/zdevito/99/head 2022-03-17T19:04:59.0549253Z * [new branch] gh/zdevito/99/orig -> origin/gh/zdevito/99/orig 2022-03-17T19:04:59.0551384Z * [new branch] gh/zhaojuanmao/100/base -> origin/gh/zhaojuanmao/100/base 2022-03-17T19:04:59.0552674Z * [new branch] gh/zhaojuanmao/100/head -> origin/gh/zhaojuanmao/100/head 2022-03-17T19:04:59.0554043Z * [new branch] gh/zhaojuanmao/100/orig -> origin/gh/zhaojuanmao/100/orig 2022-03-17T19:04:59.0555823Z * [new branch] gh/zhaojuanmao/101/base -> origin/gh/zhaojuanmao/101/base 2022-03-17T19:04:59.0557067Z * [new branch] gh/zhaojuanmao/101/head -> origin/gh/zhaojuanmao/101/head 2022-03-17T19:04:59.0558570Z * [new branch] gh/zhaojuanmao/101/orig -> origin/gh/zhaojuanmao/101/orig 2022-03-17T19:04:59.0560596Z * [new branch] gh/zhaojuanmao/5/head -> origin/gh/zhaojuanmao/5/head 2022-03-17T19:04:59.0561658Z * [new branch] gh/zhaojuanmao/5/orig -> origin/gh/zhaojuanmao/5/orig 2022-03-17T19:04:59.0563634Z * [new branch] gh/zhaojuanmao/55/base -> origin/gh/zhaojuanmao/55/base 2022-03-17T19:04:59.0564908Z * [new branch] gh/zhaojuanmao/55/head -> origin/gh/zhaojuanmao/55/head 2022-03-17T19:04:59.0566294Z * [new branch] gh/zhaojuanmao/55/orig -> origin/gh/zhaojuanmao/55/orig 2022-03-17T19:04:59.0568063Z * [new branch] gh/zhaojuanmao/61/base -> origin/gh/zhaojuanmao/61/base 2022-03-17T19:04:59.0569288Z * [new branch] gh/zhaojuanmao/61/head -> origin/gh/zhaojuanmao/61/head 2022-03-17T19:04:59.0570589Z * [new branch] gh/zhaojuanmao/61/orig -> origin/gh/zhaojuanmao/61/orig 2022-03-17T19:04:59.0572352Z * [new branch] gh/zhaojuanmao/68/base -> origin/gh/zhaojuanmao/68/base 2022-03-17T19:04:59.0573770Z * [new branch] gh/zhaojuanmao/68/head -> origin/gh/zhaojuanmao/68/head 2022-03-17T19:04:59.0575225Z * [new branch] gh/zhaojuanmao/68/orig -> origin/gh/zhaojuanmao/68/orig 2022-03-17T19:04:59.0577044Z * [new branch] gh/zhaojuanmao/69/base -> origin/gh/zhaojuanmao/69/base 2022-03-17T19:04:59.0578174Z * [new branch] gh/zhaojuanmao/69/head -> origin/gh/zhaojuanmao/69/head 2022-03-17T19:04:59.0579689Z * [new branch] gh/zhaojuanmao/69/orig -> origin/gh/zhaojuanmao/69/orig 2022-03-17T19:04:59.0581536Z * [new branch] gh/zhaojuanmao/74/base -> origin/gh/zhaojuanmao/74/base 2022-03-17T19:04:59.0582716Z * [new branch] gh/zhaojuanmao/74/head -> origin/gh/zhaojuanmao/74/head 2022-03-17T19:04:59.0584235Z * [new branch] gh/zhaojuanmao/74/orig -> origin/gh/zhaojuanmao/74/orig 2022-03-17T19:04:59.0585923Z * [new branch] gh/zhaojuanmao/80/base -> origin/gh/zhaojuanmao/80/base 2022-03-17T19:04:59.0587468Z * [new branch] gh/zhaojuanmao/80/head -> origin/gh/zhaojuanmao/80/head 2022-03-17T19:04:59.0588425Z * [new branch] gh/zhaojuanmao/80/orig -> origin/gh/zhaojuanmao/80/orig 2022-03-17T19:04:59.0590320Z * [new branch] gh/zhaojuanmao/81/base -> origin/gh/zhaojuanmao/81/base 2022-03-17T19:04:59.0591735Z * [new branch] gh/zhaojuanmao/81/head -> origin/gh/zhaojuanmao/81/head 2022-03-17T19:04:59.0593134Z * [new branch] gh/zhaojuanmao/81/orig -> origin/gh/zhaojuanmao/81/orig 2022-03-17T19:04:59.0594675Z * [new branch] gh/zhaojuanmao/82/base -> origin/gh/zhaojuanmao/82/base 2022-03-17T19:04:59.0596029Z * [new branch] gh/zhaojuanmao/82/head -> origin/gh/zhaojuanmao/82/head 2022-03-17T19:04:59.0597391Z * [new branch] gh/zhaojuanmao/82/orig -> origin/gh/zhaojuanmao/82/orig 2022-03-17T19:04:59.0599061Z * [new branch] gh/zhaojuanmao/89/base -> origin/gh/zhaojuanmao/89/base 2022-03-17T19:04:59.0600452Z * [new branch] gh/zhaojuanmao/89/head -> origin/gh/zhaojuanmao/89/head 2022-03-17T19:04:59.0602184Z * [new branch] gh/zhaojuanmao/98/base -> origin/gh/zhaojuanmao/98/base 2022-03-17T19:04:59.0604999Z * [new branch] gh/zhaojuanmao/98/head -> origin/gh/zhaojuanmao/98/head 2022-03-17T19:04:59.0606175Z * [new branch] gh/zhaojuanmao/98/orig -> origin/gh/zhaojuanmao/98/orig 2022-03-17T19:04:59.0607990Z * [new branch] gh/zhaojuanmao/99/base -> origin/gh/zhaojuanmao/99/base 2022-03-17T19:04:59.0609457Z * [new branch] gh/zhaojuanmao/99/head -> origin/gh/zhaojuanmao/99/head 2022-03-17T19:04:59.0610925Z * [new branch] gh/zhaojuanmao/99/orig -> origin/gh/zhaojuanmao/99/orig 2022-03-17T19:04:59.0612830Z * [new branch] gh/zheng-xq/1/base -> origin/gh/zheng-xq/1/base 2022-03-17T19:04:59.0614432Z * [new branch] gh/zheng-xq/1/head -> origin/gh/zheng-xq/1/head 2022-03-17T19:04:59.0615438Z * [new branch] gh/zheng-xq/1/orig -> origin/gh/zheng-xq/1/orig 2022-03-17T19:04:59.0617025Z * [new branch] gh/zheng-xq/2/base -> origin/gh/zheng-xq/2/base 2022-03-17T19:04:59.0618509Z * [new branch] gh/zheng-xq/2/head -> origin/gh/zheng-xq/2/head 2022-03-17T19:04:59.0619826Z * [new branch] gh/zheng-xq/2/orig -> origin/gh/zheng-xq/2/orig 2022-03-17T19:04:59.0621510Z * [new branch] gh/zheng-xq/4/base -> origin/gh/zheng-xq/4/base 2022-03-17T19:04:59.0622858Z * [new branch] gh/zheng-xq/4/head -> origin/gh/zheng-xq/4/head 2022-03-17T19:04:59.0623980Z * [new branch] gh/zheng-xq/4/orig -> origin/gh/zheng-xq/4/orig 2022-03-17T19:04:59.0626145Z * [new branch] gh/zhxchen17/13/base -> origin/gh/zhxchen17/13/base 2022-03-17T19:04:59.0627494Z * [new branch] gh/zhxchen17/13/head -> origin/gh/zhxchen17/13/head 2022-03-17T19:04:59.0628788Z * [new branch] gh/zhxchen17/13/orig -> origin/gh/zhxchen17/13/orig 2022-03-17T19:04:59.0630645Z * [new branch] gh/zhxchen17/14/base -> origin/gh/zhxchen17/14/base 2022-03-17T19:04:59.0631993Z * [new branch] gh/zhxchen17/14/head -> origin/gh/zhxchen17/14/head 2022-03-17T19:04:59.0633446Z * [new branch] gh/zhxchen17/15/base -> origin/gh/zhxchen17/15/base 2022-03-17T19:04:59.0634618Z * [new branch] gh/zhxchen17/15/head -> origin/gh/zhxchen17/15/head 2022-03-17T19:04:59.0636213Z * [new branch] gh/zhxchen17/16/base -> origin/gh/zhxchen17/16/base 2022-03-17T19:04:59.0637371Z * [new branch] gh/zhxchen17/16/head -> origin/gh/zhxchen17/16/head 2022-03-17T19:04:59.0638962Z * [new branch] gh/zhxchen17/17/base -> origin/gh/zhxchen17/17/base 2022-03-17T19:04:59.0640376Z * [new branch] gh/zhxchen17/17/head -> origin/gh/zhxchen17/17/head 2022-03-17T19:04:59.0641990Z * [new branch] gh/zhxchen17/18/base -> origin/gh/zhxchen17/18/base 2022-03-17T19:04:59.0643520Z * [new branch] gh/zhxchen17/18/head -> origin/gh/zhxchen17/18/head 2022-03-17T19:04:59.0644882Z * [new branch] gh/zhxchen17/19/base -> origin/gh/zhxchen17/19/base 2022-03-17T19:04:59.0646004Z * [new branch] gh/zhxchen17/19/head -> origin/gh/zhxchen17/19/head 2022-03-17T19:04:59.0647652Z * [new branch] gh/zhxchen17/20/base -> origin/gh/zhxchen17/20/base 2022-03-17T19:04:59.0648796Z * [new branch] gh/zhxchen17/20/head -> origin/gh/zhxchen17/20/head 2022-03-17T19:04:59.0650408Z * [new branch] gh/zhxchen17/21/base -> origin/gh/zhxchen17/21/base 2022-03-17T19:04:59.0651479Z * [new branch] gh/zhxchen17/21/head -> origin/gh/zhxchen17/21/head 2022-03-17T19:04:59.0653341Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2022-03-17T19:04:59.0654692Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2022-03-17T19:04:59.0655748Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2022-03-17T19:04:59.0657488Z * [new branch] gh/zhxchen17/26/base -> origin/gh/zhxchen17/26/base 2022-03-17T19:04:59.0658880Z * [new branch] gh/zhxchen17/26/orig -> origin/gh/zhxchen17/26/orig 2022-03-17T19:04:59.0660654Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2022-03-17T19:04:59.0662113Z * [new branch] gh/zhxchen17/35/orig -> origin/gh/zhxchen17/35/orig 2022-03-17T19:04:59.0663672Z * [new branch] gh/zhxchen17/36/base -> origin/gh/zhxchen17/36/base 2022-03-17T19:04:59.0665162Z * [new branch] gh/zhxchen17/36/orig -> origin/gh/zhxchen17/36/orig 2022-03-17T19:04:59.0666908Z * [new branch] gh/zhxchen17/37/base -> origin/gh/zhxchen17/37/base 2022-03-17T19:04:59.0668125Z * [new branch] gh/zhxchen17/37/orig -> origin/gh/zhxchen17/37/orig 2022-03-17T19:04:59.0669944Z * [new branch] gh/zhxchen17/38/base -> origin/gh/zhxchen17/38/base 2022-03-17T19:04:59.0671115Z * [new branch] gh/zhxchen17/38/orig -> origin/gh/zhxchen17/38/orig 2022-03-17T19:04:59.0672775Z * [new branch] gh/zhxchen17/39/base -> origin/gh/zhxchen17/39/base 2022-03-17T19:04:59.0674023Z * [new branch] gh/zhxchen17/39/orig -> origin/gh/zhxchen17/39/orig 2022-03-17T19:04:59.0675859Z * [new branch] gh/zhxchen17/4/base -> origin/gh/zhxchen17/4/base 2022-03-17T19:04:59.0679076Z * [new branch] gh/zhxchen17/4/orig -> origin/gh/zhxchen17/4/orig 2022-03-17T19:04:59.0679900Z * [new branch] gh/zhxchen17/40/base -> origin/gh/zhxchen17/40/base 2022-03-17T19:04:59.0680996Z * [new branch] gh/zhxchen17/40/orig -> origin/gh/zhxchen17/40/orig 2022-03-17T19:04:59.0682783Z * [new branch] gh/zhxchen17/41/base -> origin/gh/zhxchen17/41/base 2022-03-17T19:04:59.0684224Z * [new branch] gh/zhxchen17/41/orig -> origin/gh/zhxchen17/41/orig 2022-03-17T19:04:59.0685860Z * [new branch] gh/zhxchen17/42/base -> origin/gh/zhxchen17/42/base 2022-03-17T19:04:59.0687199Z * [new branch] gh/zhxchen17/42/orig -> origin/gh/zhxchen17/42/orig 2022-03-17T19:04:59.0688842Z * [new branch] gh/zhxchen17/69/base -> origin/gh/zhxchen17/69/base 2022-03-17T19:04:59.0690232Z * [new branch] gh/zhxchen17/69/head -> origin/gh/zhxchen17/69/head 2022-03-17T19:04:59.0691594Z * [new branch] gh/zhxchen17/69/orig -> origin/gh/zhxchen17/69/orig 2022-03-17T19:04:59.0693646Z * [new branch] gh/zhxchen17/7/base -> origin/gh/zhxchen17/7/base 2022-03-17T19:04:59.0694774Z * [new branch] gh/zhxchen17/7/head -> origin/gh/zhxchen17/7/head 2022-03-17T19:04:59.0696046Z * [new branch] gh/zhxchen17/7/orig -> origin/gh/zhxchen17/7/orig 2022-03-17T19:04:59.0697948Z * [new branch] gh/zhxchen17/79/base -> origin/gh/zhxchen17/79/base 2022-03-17T19:04:59.0699189Z * [new branch] gh/zhxchen17/79/head -> origin/gh/zhxchen17/79/head 2022-03-17T19:04:59.0700590Z * [new branch] gh/zhxchen17/79/orig -> origin/gh/zhxchen17/79/orig 2022-03-17T19:04:59.0702728Z * [new branch] gh/zmvictor/1/base -> origin/gh/zmvictor/1/base 2022-03-17T19:04:59.0703951Z * [new branch] gh/zmvictor/1/head -> origin/gh/zmvictor/1/head 2022-03-17T19:04:59.0706084Z * [new branch] gh/zou3519/117/base -> origin/gh/zou3519/117/base 2022-03-17T19:04:59.0707366Z * [new branch] gh/zou3519/117/head -> origin/gh/zou3519/117/head 2022-03-17T19:04:59.0708775Z * [new branch] gh/zou3519/117/orig -> origin/gh/zou3519/117/orig 2022-03-17T19:04:59.0710169Z * [new branch] gh/zou3519/118/base -> origin/gh/zou3519/118/base 2022-03-17T19:04:59.0711541Z * [new branch] gh/zou3519/118/head -> origin/gh/zou3519/118/head 2022-03-17T19:04:59.0712985Z * [new branch] gh/zou3519/118/orig -> origin/gh/zou3519/118/orig 2022-03-17T19:04:59.0714504Z * [new branch] gh/zou3519/119/base -> origin/gh/zou3519/119/base 2022-03-17T19:04:59.0715975Z * [new branch] gh/zou3519/119/head -> origin/gh/zou3519/119/head 2022-03-17T19:04:59.0717312Z * [new branch] gh/zou3519/119/orig -> origin/gh/zou3519/119/orig 2022-03-17T19:04:59.0718802Z * [new branch] gh/zou3519/122/base -> origin/gh/zou3519/122/base 2022-03-17T19:04:59.0720265Z * [new branch] gh/zou3519/122/head -> origin/gh/zou3519/122/head 2022-03-17T19:04:59.0722067Z * [new branch] gh/zou3519/122/orig -> origin/gh/zou3519/122/orig 2022-03-17T19:04:59.0724814Z * [new branch] gh/zou3519/144/base -> origin/gh/zou3519/144/base 2022-03-17T19:04:59.0725963Z * [new branch] gh/zou3519/144/head -> origin/gh/zou3519/144/head 2022-03-17T19:04:59.0727265Z * [new branch] gh/zou3519/144/orig -> origin/gh/zou3519/144/orig 2022-03-17T19:04:59.0729276Z * [new branch] gh/zou3519/155/head -> origin/gh/zou3519/155/head 2022-03-17T19:04:59.0730474Z * [new branch] gh/zou3519/155/orig -> origin/gh/zou3519/155/orig 2022-03-17T19:04:59.0732431Z * [new branch] gh/zou3519/162/head -> origin/gh/zou3519/162/head 2022-03-17T19:04:59.0733661Z * [new branch] gh/zou3519/162/orig -> origin/gh/zou3519/162/orig 2022-03-17T19:04:59.0735320Z * [new branch] gh/zou3519/207/base -> origin/gh/zou3519/207/base 2022-03-17T19:04:59.0736808Z * [new branch] gh/zou3519/207/head -> origin/gh/zou3519/207/head 2022-03-17T19:04:59.0738229Z * [new branch] gh/zou3519/207/orig -> origin/gh/zou3519/207/orig 2022-03-17T19:04:59.0739673Z * [new branch] gh/zou3519/208/base -> origin/gh/zou3519/208/base 2022-03-17T19:04:59.0741290Z * [new branch] gh/zou3519/208/head -> origin/gh/zou3519/208/head 2022-03-17T19:04:59.0742763Z * [new branch] gh/zou3519/208/orig -> origin/gh/zou3519/208/orig 2022-03-17T19:04:59.0744401Z * [new branch] gh/zou3519/219/base -> origin/gh/zou3519/219/base 2022-03-17T19:04:59.0746287Z * [new branch] gh/zou3519/219/head -> origin/gh/zou3519/219/head 2022-03-17T19:04:59.0747771Z * [new branch] gh/zou3519/219/orig -> origin/gh/zou3519/219/orig 2022-03-17T19:04:59.0749612Z * [new branch] gh/zou3519/224/base -> origin/gh/zou3519/224/base 2022-03-17T19:04:59.0750809Z * [new branch] gh/zou3519/224/head -> origin/gh/zou3519/224/head 2022-03-17T19:04:59.0752036Z * [new branch] gh/zou3519/224/orig -> origin/gh/zou3519/224/orig 2022-03-17T19:04:59.0753852Z * [new branch] gh/zou3519/251/base -> origin/gh/zou3519/251/base 2022-03-17T19:04:59.0755173Z * [new branch] gh/zou3519/251/head -> origin/gh/zou3519/251/head 2022-03-17T19:04:59.0756702Z * [new branch] gh/zou3519/251/orig -> origin/gh/zou3519/251/orig 2022-03-17T19:04:59.0758212Z * [new branch] gh/zou3519/252/base -> origin/gh/zou3519/252/base 2022-03-17T19:04:59.0759835Z * [new branch] gh/zou3519/252/head -> origin/gh/zou3519/252/head 2022-03-17T19:04:59.0761490Z * [new branch] gh/zou3519/252/orig -> origin/gh/zou3519/252/orig 2022-03-17T19:04:59.0763499Z * [new branch] gh/zou3519/277/base -> origin/gh/zou3519/277/base 2022-03-17T19:04:59.0764878Z * [new branch] gh/zou3519/277/head -> origin/gh/zou3519/277/head 2022-03-17T19:04:59.0766163Z * [new branch] gh/zou3519/277/orig -> origin/gh/zou3519/277/orig 2022-03-17T19:04:59.0768074Z * [new branch] gh/zou3519/284/base -> origin/gh/zou3519/284/base 2022-03-17T19:04:59.0769356Z * [new branch] gh/zou3519/284/head -> origin/gh/zou3519/284/head 2022-03-17T19:04:59.0770644Z * [new branch] gh/zou3519/284/orig -> origin/gh/zou3519/284/orig 2022-03-17T19:04:59.0772506Z * [new branch] gh/zou3519/293/base -> origin/gh/zou3519/293/base 2022-03-17T19:04:59.0773430Z * [new branch] gh/zou3519/293/head -> origin/gh/zou3519/293/head 2022-03-17T19:04:59.0775319Z * [new branch] gh/zou3519/293/orig -> origin/gh/zou3519/293/orig 2022-03-17T19:04:59.0776945Z * [new branch] gh/zou3519/298/base -> origin/gh/zou3519/298/base 2022-03-17T19:04:59.0778528Z * [new branch] gh/zou3519/298/head -> origin/gh/zou3519/298/head 2022-03-17T19:04:59.0779538Z * [new branch] gh/zou3519/298/orig -> origin/gh/zou3519/298/orig 2022-03-17T19:04:59.0781228Z * [new branch] gh/zou3519/308/base -> origin/gh/zou3519/308/base 2022-03-17T19:04:59.0782661Z * [new branch] gh/zou3519/308/head -> origin/gh/zou3519/308/head 2022-03-17T19:04:59.0784151Z * [new branch] gh/zou3519/308/orig -> origin/gh/zou3519/308/orig 2022-03-17T19:04:59.0785869Z * [new branch] gh/zou3519/311/base -> origin/gh/zou3519/311/base 2022-03-17T19:04:59.0787485Z * [new branch] gh/zou3519/311/head -> origin/gh/zou3519/311/head 2022-03-17T19:04:59.0789003Z * [new branch] gh/zou3519/311/orig -> origin/gh/zou3519/311/orig 2022-03-17T19:04:59.0790448Z * [new branch] gh/zou3519/312/base -> origin/gh/zou3519/312/base 2022-03-17T19:04:59.0792024Z * [new branch] gh/zou3519/312/head -> origin/gh/zou3519/312/head 2022-03-17T19:04:59.0793390Z * [new branch] gh/zou3519/312/orig -> origin/gh/zou3519/312/orig 2022-03-17T19:04:59.0795088Z * [new branch] gh/zou3519/313/base -> origin/gh/zou3519/313/base 2022-03-17T19:04:59.0796566Z * [new branch] gh/zou3519/313/head -> origin/gh/zou3519/313/head 2022-03-17T19:04:59.0798125Z * [new branch] gh/zou3519/313/orig -> origin/gh/zou3519/313/orig 2022-03-17T19:04:59.0799841Z * [new branch] gh/zou3519/339/base -> origin/gh/zou3519/339/base 2022-03-17T19:04:59.0801413Z * [new branch] gh/zou3519/339/head -> origin/gh/zou3519/339/head 2022-03-17T19:04:59.0803918Z * [new branch] gh/zou3519/339/orig -> origin/gh/zou3519/339/orig 2022-03-17T19:04:59.0805547Z * [new branch] gh/zou3519/348/base -> origin/gh/zou3519/348/base 2022-03-17T19:04:59.0806958Z * [new branch] gh/zou3519/348/head -> origin/gh/zou3519/348/head 2022-03-17T19:04:59.0808296Z * [new branch] gh/zou3519/348/orig -> origin/gh/zou3519/348/orig 2022-03-17T19:04:59.0810032Z * [new branch] gh/zou3519/351/base -> origin/gh/zou3519/351/base 2022-03-17T19:04:59.0811155Z * [new branch] gh/zou3519/351/head -> origin/gh/zou3519/351/head 2022-03-17T19:04:59.0812589Z * [new branch] gh/zou3519/351/orig -> origin/gh/zou3519/351/orig 2022-03-17T19:04:59.0815024Z * [new branch] gh/zou3519/382/base -> origin/gh/zou3519/382/base 2022-03-17T19:04:59.0816358Z * [new branch] gh/zou3519/382/head -> origin/gh/zou3519/382/head 2022-03-17T19:04:59.0817664Z * [new branch] gh/zou3519/382/orig -> origin/gh/zou3519/382/orig 2022-03-17T19:04:59.0819358Z * [new branch] gh/zou3519/387/base -> origin/gh/zou3519/387/base 2022-03-17T19:04:59.0820729Z * [new branch] gh/zou3519/387/head -> origin/gh/zou3519/387/head 2022-03-17T19:04:59.0822092Z * [new branch] gh/zou3519/387/orig -> origin/gh/zou3519/387/orig 2022-03-17T19:04:59.0823847Z * [new branch] gh/zou3519/400/base -> origin/gh/zou3519/400/base 2022-03-17T19:04:59.0825190Z * [new branch] gh/zou3519/400/head -> origin/gh/zou3519/400/head 2022-03-17T19:04:59.0826545Z * [new branch] gh/zou3519/400/orig -> origin/gh/zou3519/400/orig 2022-03-17T19:04:59.0828230Z * [new branch] gh/zou3519/403/base -> origin/gh/zou3519/403/base 2022-03-17T19:04:59.0829582Z * [new branch] gh/zou3519/403/head -> origin/gh/zou3519/403/head 2022-03-17T19:04:59.0830935Z * [new branch] gh/zou3519/403/orig -> origin/gh/zou3519/403/orig 2022-03-17T19:04:59.0832651Z * [new branch] gh/zou3519/404/base -> origin/gh/zou3519/404/base 2022-03-17T19:04:59.0834316Z * [new branch] gh/zou3519/404/head -> origin/gh/zou3519/404/head 2022-03-17T19:04:59.0835399Z * [new branch] gh/zou3519/404/orig -> origin/gh/zou3519/404/orig 2022-03-17T19:04:59.0837180Z * [new branch] gh/zou3519/408/base -> origin/gh/zou3519/408/base 2022-03-17T19:04:59.0838526Z * [new branch] gh/zou3519/408/head -> origin/gh/zou3519/408/head 2022-03-17T19:04:59.0839934Z * [new branch] gh/zou3519/408/orig -> origin/gh/zou3519/408/orig 2022-03-17T19:04:59.0841684Z * [new branch] gh/zou3519/411/base -> origin/gh/zou3519/411/base 2022-03-17T19:04:59.0843313Z * [new branch] gh/zou3519/411/head -> origin/gh/zou3519/411/head 2022-03-17T19:04:59.0844743Z * [new branch] gh/zou3519/411/orig -> origin/gh/zou3519/411/orig 2022-03-17T19:04:59.0846448Z * [new branch] gh/zou3519/412/base -> origin/gh/zou3519/412/base 2022-03-17T19:04:59.0848051Z * [new branch] gh/zou3519/412/head -> origin/gh/zou3519/412/head 2022-03-17T19:04:59.0849456Z * [new branch] gh/zou3519/412/orig -> origin/gh/zou3519/412/orig 2022-03-17T19:04:59.0851174Z * [new branch] gh/zou3519/413/base -> origin/gh/zou3519/413/base 2022-03-17T19:04:59.0852570Z * [new branch] gh/zou3519/413/head -> origin/gh/zou3519/413/head 2022-03-17T19:04:59.0853681Z * [new branch] gh/zou3519/413/orig -> origin/gh/zou3519/413/orig 2022-03-17T19:04:59.0855491Z * [new branch] gh/zou3519/414/base -> origin/gh/zou3519/414/base 2022-03-17T19:04:59.0856893Z * [new branch] gh/zou3519/414/head -> origin/gh/zou3519/414/head 2022-03-17T19:04:59.0858244Z * [new branch] gh/zou3519/414/orig -> origin/gh/zou3519/414/orig 2022-03-17T19:04:59.0860067Z * [new branch] gh/zou3519/415/base -> origin/gh/zou3519/415/base 2022-03-17T19:04:59.0861214Z * [new branch] gh/zou3519/415/head -> origin/gh/zou3519/415/head 2022-03-17T19:04:59.0862713Z * [new branch] gh/zou3519/415/orig -> origin/gh/zou3519/415/orig 2022-03-17T19:04:59.0864622Z * [new branch] gh/zou3519/87/base -> origin/gh/zou3519/87/base 2022-03-17T19:04:59.0866135Z * [new branch] gh/zou3519/87/head -> origin/gh/zou3519/87/head 2022-03-17T19:04:59.0867662Z * [new branch] gh/zou3519/87/orig -> origin/gh/zou3519/87/orig 2022-03-17T19:04:59.0869670Z * [new branch] gh/zzzwen/1/base -> origin/gh/zzzwen/1/base 2022-03-17T19:04:59.0871170Z * [new branch] gh/zzzwen/1/orig -> origin/gh/zzzwen/1/orig 2022-03-17T19:04:59.0872735Z * [new branch] gh/zzzwen/2/base -> origin/gh/zzzwen/2/base 2022-03-17T19:04:59.0874074Z * [new branch] gh/zzzwen/2/head -> origin/gh/zzzwen/2/head 2022-03-17T19:04:59.0875468Z * [new branch] gh/zzzwen/2/orig -> origin/gh/zzzwen/2/orig 2022-03-17T19:04:59.0876996Z * [new branch] git_submodule_docs -> origin/git_submodule_docs 2022-03-17T19:04:59.0878670Z * [new branch] gj/Ivan -> origin/gj/Ivan 2022-03-17T19:04:59.0880162Z * [new branch] gradle_bump -> origin/gradle_bump 2022-03-17T19:04:59.0881493Z * [new branch] hagiopol -> origin/hagiopol 2022-03-17T19:04:59.0883696Z * [new branch] hcz_debug -> origin/hcz_debug 2022-03-17T19:04:59.0885802Z * [new branch] heitorschueroff/einsum/optimize -> origin/heitorschueroff/einsum/optimize 2022-03-17T19:04:59.0886968Z * [new branch] heitorschueroff/minmax -> origin/heitorschueroff/minmax 2022-03-17T19:04:59.0888703Z * [new branch] heitorschueroff/opinfo/fails -> origin/heitorschueroff/opinfo/fails 2022-03-17T19:04:59.0890118Z * [new branch] heitorschueroff/opinfo/reductions -> origin/heitorschueroff/opinfo/reductions 2022-03-17T19:04:59.0891845Z * [new branch] heitorschueroff/quantile/interpolation -> origin/heitorschueroff/quantile/interpolation 2022-03-17T19:04:59.0893198Z * [new branch] helper -> origin/helper 2022-03-17T19:04:59.0894731Z * [new branch] hide_internal_API -> origin/hide_internal_API 2022-03-17T19:04:59.0895869Z * [new branch] hide_set_enabled -> origin/hide_set_enabled 2022-03-17T19:04:59.0897136Z * [new branch] histc_docs_fix -> origin/histc_docs_fix 2022-03-17T19:04:59.0898819Z * [new branch] histc_opinfo -> origin/histc_opinfo 2022-03-17T19:04:59.0899930Z * [new branch] histc_perf_fix -> origin/histc_perf_fix 2022-03-17T19:04:59.0901333Z * [new branch] histogramdd -> origin/histogramdd 2022-03-17T19:04:59.0902661Z * [new branch] histogramdd_rsts -> origin/histogramdd_rsts 2022-03-17T19:04:59.0904079Z * [new branch] holly1238-patch-1 -> origin/holly1238-patch-1 2022-03-17T19:04:59.0905461Z * [new branch] hot-fix-librosa -> origin/hot-fix-librosa 2022-03-17T19:04:59.0906903Z * [new branch] hotfix_packaging_version -> origin/hotfix_packaging_version 2022-03-17T19:04:59.0908206Z * [new branch] hub_improvements -> origin/hub_improvements 2022-03-17T19:04:59.0909612Z * [new branch] hub_security -> origin/hub_security 2022-03-17T19:04:59.0910940Z * [new branch] hufsb -> origin/hufsb 2022-03-17T19:04:59.0912538Z * [new branch] igorsyl/1 -> origin/igorsyl/1 2022-03-17T19:04:59.0914005Z * [new branch] ik-test -> origin/ik-test 2022-03-17T19:04:59.0915582Z * [new branch] ik_android_byte_tensors_tests -> origin/ik_android_byte_tensors_tests 2022-03-17T19:04:59.0917038Z * [new branch] ik_android_generic_dict_issue -> origin/ik_android_generic_dict_issue 2022-03-17T19:04:59.0918545Z * [new branch] ik_android_gradle_bintray_publishing -> origin/ik_android_gradle_bintray_publishing 2022-03-17T19:04:59.0919647Z * [new branch] ik_android_vision_libyuv -> origin/ik_android_vision_libyuv 2022-03-17T19:04:59.0920997Z * [new branch] ik_lapack_qml -> origin/ik_lapack_qml 2022-03-17T19:04:59.0922443Z * [new branch] ik_vulkan_ci -> origin/ik_vulkan_ci 2022-03-17T19:04:59.0924033Z * [new branch] ik_wip_android_stdouterr_to_logcat -> origin/ik_wip_android_stdouterr_to_logcat 2022-03-17T19:04:59.0925265Z * [new branch] illegal_memory_access -> origin/illegal_memory_access 2022-03-17T19:04:59.0926404Z * [new branch] independent-constraint -> origin/independent-constraint 2022-03-17T19:04:59.0927789Z * [new branch] independent-transform -> origin/independent-transform 2022-03-17T19:04:59.0929175Z * [new branch] infer_types -> origin/infer_types 2022-03-17T19:04:59.0930806Z * [new branch] infer_types_methods -> origin/infer_types_methods 2022-03-17T19:04:59.0932282Z * [new branch] informative-dist-support-error -> origin/informative-dist-support-error 2022-03-17T19:04:59.0933371Z * [new branch] inplaceorview-test-ci -> origin/inplaceorview-test-ci 2022-03-17T19:04:59.0935301Z * [new branch] iotamudelta/update_rocm3.1 -> origin/iotamudelta/update_rocm3.1 2022-03-17T19:04:59.0936434Z * [new branch] isin-feature -> origin/isin-feature 2022-03-17T19:04:59.0937830Z * [new branch] issue#58739 -> origin/issue#58739 2022-03-17T19:04:59.0939188Z * [new branch] issue6009 -> origin/issue6009 2022-03-17T19:04:59.0941104Z * [new branch] ivan/add-symbol-qr -> origin/ivan/add-symbol-qr 2022-03-17T19:04:59.0942258Z * [new branch] jamesr66a-patch-1 -> origin/jamesr66a-patch-1 2022-03-17T19:04:59.0943658Z * [new branch] jamesr66a-patch-2 -> origin/jamesr66a-patch-2 2022-03-17T19:04:59.0945187Z * [new branch] jenkins_and_tools_update_master_to_main -> origin/jenkins_and_tools_update_master_to_main 2022-03-17T19:04:59.0946419Z * [new branch] jerryzh168-patch-6 -> origin/jerryzh168-patch-6 2022-03-17T19:04:59.0947683Z * [new branch] jit-language-reference-types-ii -> origin/jit-language-reference-types-ii 2022-03-17T19:04:59.0948990Z * [new branch] jit-language-spec -> origin/jit-language-spec 2022-03-17T19:04:59.0950561Z * [new branch] jit3.9.1 -> origin/jit3.9.1 2022-03-17T19:04:59.0951913Z * [new branch] jit_future_overview_fix -> origin/jit_future_overview_fix 2022-03-17T19:04:59.0953285Z * [new branch] jiterator_documentation -> origin/jiterator_documentation 2022-03-17T19:04:59.0954632Z * [new branch] jlin27-docs-toc -> origin/jlin27-docs-toc 2022-03-17T19:04:59.0956137Z * [new branch] jlin27-sphinx-tag -> origin/jlin27-sphinx-tag 2022-03-17T19:04:59.0957553Z * [new branch] jlin27-sphinx-tag-2 -> origin/jlin27-sphinx-tag-2 2022-03-17T19:04:59.0958962Z * [new branch] jspisak-patch-1 -> origin/jspisak-patch-1 2022-03-17T19:04:59.0960337Z * [new branch] jspisak-patch-2 -> origin/jspisak-patch-2 2022-03-17T19:04:59.0961727Z * [new branch] jspisak-patch-3 -> origin/jspisak-patch-3 2022-03-17T19:04:59.0963663Z * [new branch] kerryz/test-ci -> origin/kerryz/test-ci 2022-03-17T19:04:59.0964942Z * [new branch] kill_mixin -> origin/kill_mixin 2022-03-17T19:04:59.0966325Z * [new branch] kineto_exp -> origin/kineto_exp 2022-03-17T19:04:59.0967755Z * [new branch] kineto_metadata -> origin/kineto_metadata 2022-03-17T19:04:59.0969560Z * [new branch] kiuk/cp-66182 -> origin/kiuk/cp-66182 2022-03-17T19:04:59.0971499Z * [new branch] krovatkin/bench_timings -> origin/krovatkin/bench_timings 2022-03-17T19:04:59.0972620Z * [new branch] krovatkin/conv2d_out -> origin/krovatkin/conv2d_out 2022-03-17T19:04:59.0974020Z * [new branch] krovatkin/cse_changed -> origin/krovatkin/cse_changed 2022-03-17T19:04:59.0975307Z * [new branch] krovatkin/dummy_torchbench -> origin/krovatkin/dummy_torchbench 2022-03-17T19:04:59.0976595Z * [new branch] krovatkin/gamrix_25 -> origin/krovatkin/gamrix_25 2022-03-17T19:04:59.0978085Z * [new branch] krovatkin/improve_error -> origin/krovatkin/improve_error 2022-03-17T19:04:59.0979054Z * [new branch] krovatkin/inline_threshold -> origin/krovatkin/inline_threshold 2022-03-17T19:04:59.0980517Z * [new branch] krovatkin/inline_threshold2 -> origin/krovatkin/inline_threshold2 2022-03-17T19:04:59.0981622Z * [new branch] krovatkin/kevin_assert -> origin/krovatkin/kevin_assert 2022-03-17T19:04:59.0982957Z * [new branch] krovatkin/opt_for_infer -> origin/krovatkin/opt_for_infer 2022-03-17T19:04:59.0984111Z * [new branch] krovatkin/opt_for_infr -> origin/krovatkin/opt_for_infr 2022-03-17T19:04:59.0985432Z * [new branch] krovatkin/opt_for_infr2 -> origin/krovatkin/opt_for_infr2 2022-03-17T19:04:59.0986816Z * [new branch] krovatkin/reformat -> origin/krovatkin/reformat 2022-03-17T19:04:59.0988174Z * [new branch] krovatkin/squeeze -> origin/krovatkin/squeeze 2022-03-17T19:04:59.0989543Z * [new branch] krovatkin/squeeze_master -> origin/krovatkin/squeeze_master 2022-03-17T19:04:59.0990694Z * [new branch] krovatkin/sub_lazy_bench -> origin/krovatkin/sub_lazy_bench 2022-03-17T19:04:59.0992094Z * [new branch] krovatkin/test_legacy_fuser -> origin/krovatkin/test_legacy_fuser 2022-03-17T19:04:59.0993426Z * [new branch] krovatkin/unsqueeze_ltc -> origin/krovatkin/unsqueeze_ltc 2022-03-17T19:04:59.0995049Z * [new branch] ktse/codegen_exercises -> origin/ktse/codegen_exercises 2022-03-17T19:04:59.0996780Z * [new branch] lan/test_repo_dispatch_circleci -> origin/lan/test_repo_dispatch_circleci 2022-03-17T19:04:59.0997982Z * [new branch] lan/test_repo_dispatch_gha -> origin/lan/test_repo_dispatch_gha 2022-03-17T19:04:59.0999365Z * [new branch] lan/test_reverse_trigger_binary -> origin/lan/test_reverse_trigger_binary 2022-03-17T19:04:59.1000746Z * [new branch] larryliu-scratch -> origin/larryliu-scratch 2022-03-17T19:04:59.1002172Z * [new branch] layer_norm_tbb -> origin/layer_norm_tbb 2022-03-17T19:04:59.1003789Z * [new branch] lazy_tensor_staging -> origin/lazy_tensor_staging 2022-03-17T19:04:59.1004888Z * [new branch] libcudacxx -> origin/libcudacxx 2022-03-17T19:04:59.1006301Z * [new branch] linalg_docs -> origin/linalg_docs 2022-03-17T19:04:59.1007850Z * [new branch] linalg_svd -> origin/linalg_svd 2022-03-17T19:04:59.1009307Z * [new branch] linbinyu-patch-1 -> origin/linbinyu-patch-1 2022-03-17T19:04:59.1010636Z * [new branch] linbinyu-patch-2 -> origin/linbinyu-patch-2 2022-03-17T19:04:59.1011962Z * [new branch] linear_dyn -> origin/linear_dyn 2022-03-17T19:04:59.1013353Z * [new branch] linear_dynamic -> origin/linear_dynamic 2022-03-17T19:04:59.1014770Z * [new branch] linear_qat_lowering -> origin/linear_qat_lowering 2022-03-17T19:04:59.1015988Z * [new branch] lint2 -> origin/lint2 2022-03-17T19:04:59.1017420Z * [new branch] llvmconda -> origin/llvmconda 2022-03-17T19:04:59.1018756Z * [new branch] llvmpip -> origin/llvmpip 2022-03-17T19:04:59.1020129Z * [new branch] load-ucc.so -> origin/load-ucc.so 2022-03-17T19:04:59.1021542Z * [new branch] logcumsumexp -> origin/logcumsumexp 2022-03-17T19:04:59.1022918Z * [new branch] ltc_threshold -> origin/ltc_threshold 2022-03-17T19:04:59.1025058Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2022-03-17T19:04:59.1026409Z * [new branch] lw_c10d_debug_fix -> origin/lw_c10d_debug_fix 2022-03-17T19:04:59.1027786Z * [new branch] macos-conda-llvm -> origin/macos-conda-llvm 2022-03-17T19:04:59.1029129Z * [new branch] main -> origin/main 2022-03-17T19:04:59.1030752Z * [new branch] makslevental-patch-1 -> origin/makslevental-patch-1 2022-03-17T19:04:59.1032197Z * [new branch] makslevental-patch-1-1 -> origin/makslevental-patch-1-1 2022-03-17T19:04:59.1033593Z * [new branch] makslevental-patch-2 -> origin/makslevental-patch-2 2022-03-17T19:04:59.1034942Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2022-03-17T19:04:59.1036380Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2022-03-17T19:04:59.1037769Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2022-03-17T19:04:59.1039192Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2022-03-17T19:04:59.1040528Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2022-03-17T19:04:59.1042213Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2022-03-17T19:04:59.1044388Z * [new branch] malfet/add-lint-as-mandatory-check-for-ghf -> origin/malfet/add-lint-as-mandatory-check-for-ghf 2022-03-17T19:04:59.1045574Z * [new branch] malfet/add-merge-rules -> origin/malfet/add-merge-rules 2022-03-17T19:04:59.1047430Z * [new branch] malfet/clang-tidy-fixes-take-3 -> origin/malfet/clang-tidy-fixes-take-3 2022-03-17T19:04:59.1048416Z * [new branch] malfet/cp-lazy-load -> origin/malfet/cp-lazy-load 2022-03-17T19:04:59.1049936Z * [new branch] malfet/dedup-cpp-aot-test-names -> origin/malfet/dedup-cpp-aot-test-names 2022-03-17T19:04:59.1051262Z * [new branch] malfet/disable-mac-smoke-builds -> origin/malfet/disable-mac-smoke-builds 2022-03-17T19:04:59.1052551Z * [new branch] malfet/fix-asmjit-clang-13-werror -> origin/malfet/fix-asmjit-clang-13-werror 2022-03-17T19:04:59.1055538Z * [new branch] malfet/ghf-add-push-retries -> origin/malfet/ghf-add-push-retries 2022-03-17T19:04:59.1056799Z * [new branch] malfet/leak-python-engine -> origin/malfet/leak-python-engine 2022-03-17T19:04:59.1057831Z * [new branch] malfet/make-dispatch-globals-const -> origin/malfet/make-dispatch-globals-const 2022-03-17T19:04:59.1058981Z * [new branch] malfet/move-cublas-dependency-after-cudnn -> origin/malfet/move-cublas-dependency-after-cudnn 2022-03-17T19:04:59.1059946Z * [new branch] malfet/refactor-GloomakeDeviceFor -> origin/malfet/refactor-GloomakeDeviceFor 2022-03-17T19:04:59.1060879Z * [new branch] malfet/split-cuda-spectral-ops -> origin/malfet/split-cuda-spectral-ops 2022-03-17T19:04:59.1062149Z * [new branch] malfet/test-builder-changes -> origin/malfet/test-builder-changes 2022-03-17T19:04:59.1063318Z * [new branch] malfet/test-builder-m1 -> origin/malfet/test-builder-m1 2022-03-17T19:04:59.1064900Z * [new branch] malfet/try-enabling-AVX2-on-Windows -> origin/malfet/try-enabling-AVX2-on-Windows 2022-03-17T19:04:59.1066134Z * [new branch] malfet/unify-workflow-dispatch -> origin/malfet/unify-workflow-dispatch 2022-03-17T19:04:59.1067554Z * [new branch] malfet/use-std-copy-in-cat-kernel -> origin/malfet/use-std-copy-in-cat-kernel 2022-03-17T19:04:59.1068680Z * [new branch] mark-test-hotfix -> origin/mark-test-hotfix 2022-03-17T19:04:59.1070018Z * [new branch] master -> origin/master 2022-03-17T19:04:59.1071366Z * [new branch] mathtests -> origin/mathtests 2022-03-17T19:04:59.1072745Z * [new branch] matmul_vectors -> origin/matmul_vectors 2022-03-17T19:04:59.1074141Z * [new branch] matrix_exp -> origin/matrix_exp 2022-03-17T19:04:59.1075532Z * [new branch] max_pool_docs -> origin/max_pool_docs 2022-03-17T19:04:59.1076958Z * [new branch] measure_warmup -> origin/measure_warmup 2022-03-17T19:04:59.1078635Z * [new branch] meganset/output_ratio -> origin/meganset/output_ratio 2022-03-17T19:04:59.1079877Z * [new branch] mem_docs -> origin/mem_docs 2022-03-17T19:04:59.1081175Z * [new branch] merge -> origin/merge 2022-03-17T19:04:59.1092606Z * [new branch] merge_cuda_streams -> origin/merge_cuda_streams 2022-03-17T19:04:59.1093467Z * [new branch] merge_eager_jit_cuda_streams -> origin/merge_eager_jit_cuda_streams 2022-03-17T19:04:59.1094251Z * [new branch] merge_script_pdt_api -> origin/merge_script_pdt_api 2022-03-17T19:04:59.1095017Z * [new branch] meta_support_yf225 -> origin/meta_support_yf225 2022-03-17T19:04:59.1096304Z * [new branch] metacompile_ternary_if -> origin/metacompile_ternary_if 2022-03-17T19:04:59.1097945Z * [new branch] mgpu_integration_proto_ops -> origin/mgpu_integration_proto_ops 2022-03-17T19:04:59.1098982Z * [new branch] mgpu_integration_proto_tex -> origin/mgpu_integration_proto_tex 2022-03-17T19:04:59.1100593Z * [new branch] migrate_other_documentation_from_master_to_main -> origin/migrate_other_documentation_from_master_to_main 2022-03-17T19:04:59.1101923Z * [new branch] migrate_torch_master_references_to_main -> origin/migrate_torch_master_references_to_main 2022-03-17T19:04:59.1102962Z * [new branch] mingfeima -> origin/mingfeima 2022-03-17T19:04:59.1104505Z * [new branch] mobile_cpu_perf_benchmark_pr -> origin/mobile_cpu_perf_benchmark_pr 2022-03-17T19:04:59.1105889Z * [new branch] monitored_barrier_fix -> origin/monitored_barrier_fix 2022-03-17T19:04:59.1107096Z * [new branch] monkey_type -> origin/monkey_type 2022-03-17T19:04:59.1108739Z * [new branch] mostafaelhoushi-patch-1 -> origin/mostafaelhoushi-patch-1 2022-03-17T19:04:59.1110672Z * [new branch] mostafaelhoushi-patch-fix-wrap-fn-in-ast-rewriter -> origin/mostafaelhoushi-patch-fix-wrap-fn-in-ast-rewriter 2022-03-17T19:04:59.1111561Z * [new branch] mrshenli-patch-1 -> origin/mrshenli-patch-1 2022-03-17T19:04:59.1113014Z * [new branch] mrshenli-patch-2 -> origin/mrshenli-patch-2 2022-03-17T19:04:59.1114249Z * [new branch] multi_output_0 -> origin/multi_output_0 2022-03-17T19:04:59.1115502Z * [new branch] multinomial_half -> origin/multinomial_half 2022-03-17T19:04:59.1116965Z * [new branch] mvz-nnc-aot-prototype -> origin/mvz-nnc-aot-prototype 2022-03-17T19:04:59.1118258Z * [new branch] mvz-te-cat-fix-1.7 -> origin/mvz-te-cat-fix-1.7 2022-03-17T19:04:59.1119489Z * [new branch] nccl-on-cpu -> origin/nccl-on-cpu 2022-03-17T19:04:59.1120853Z * [new branch] nestedtensor -> origin/nestedtensor 2022-03-17T19:04:59.1122405Z * [new branch] new_add_default_args -> origin/new_add_default_args 2022-03-17T19:04:59.1123838Z * [new branch] new_merge_streams_1 -> origin/new_merge_streams_1 2022-03-17T19:04:59.1125199Z * [new branch] new_metacompile_if -> origin/new_metacompile_if 2022-03-17T19:04:59.1126463Z * [new branch] new_ones -> origin/new_ones 2022-03-17T19:04:59.1128021Z * [new branch] new_script_type -> origin/new_script_type 2022-03-17T19:04:59.1129364Z * [new branch] newest_ut -> origin/newest_ut 2022-03-17T19:04:59.1131550Z * [new branch] ngimel/THCTensor_cleanup -> origin/ngimel/THCTensor_cleanup 2022-03-17T19:04:59.1132628Z * [new branch] ngimel/THC_reduce_cleanup -> origin/ngimel/THC_reduce_cleanup 2022-03-17T19:04:59.1133874Z * [new branch] ngimel/addmm_instr -> origin/ngimel/addmm_instr 2022-03-17T19:04:59.1135166Z * [new branch] ngimel/addr_test -> origin/ngimel/addr_test 2022-03-17T19:04:59.1136480Z * [new branch] ngimel/alias_gen -> origin/ngimel/alias_gen 2022-03-17T19:04:59.1137738Z * [new branch] ngimel/cleanup -> origin/ngimel/cleanup 2022-03-17T19:04:59.1139047Z * [new branch] ngimel/conv_db -> origin/ngimel/conv_db 2022-03-17T19:04:59.1140336Z * [new branch] ngimel/cos_sim -> origin/ngimel/cos_sim 2022-03-17T19:04:59.1141681Z * [new branch] ngimel/cov_errors -> origin/ngimel/cov_errors 2022-03-17T19:04:59.1143013Z * [new branch] ngimel/fix_indexing -> origin/ngimel/fix_indexing 2022-03-17T19:04:59.1144255Z * [new branch] ngimel/fix_unique -> origin/ngimel/fix_unique 2022-03-17T19:04:59.1146021Z * [new branch] ngimel/flip_resubmit -> origin/ngimel/flip_resubmit 2022-03-17T19:04:59.1146950Z * [new branch] ngimel/i0_accscalar -> origin/ngimel/i0_accscalar 2022-03-17T19:04:59.1148225Z * [new branch] ngimel/igamma -> origin/ngimel/igamma 2022-03-17T19:04:59.1149504Z * [new branch] ngimel/launch_check -> origin/ngimel/launch_check 2022-03-17T19:04:59.1150769Z * [new branch] ngimel/layer_norm -> origin/ngimel/layer_norm 2022-03-17T19:04:59.1152115Z * [new branch] ngimel/linalg_docs -> origin/ngimel/linalg_docs 2022-03-17T19:04:59.1153382Z * [new branch] ngimel/loops_reduce -> origin/ngimel/loops_reduce 2022-03-17T19:04:59.1154692Z * [new branch] ngimel/make_tensor_grad -> origin/ngimel/make_tensor_grad 2022-03-17T19:04:59.1155891Z * [new branch] ngimel/map_alloc_rand -> origin/ngimel/map_alloc_rand 2022-03-17T19:04:59.1157161Z * [new branch] ngimel/masked_select -> origin/ngimel/masked_select 2022-03-17T19:04:59.1158528Z * [new branch] ngimel/no_copy_indices -> origin/ngimel/no_copy_indices 2022-03-17T19:04:59.1159705Z * [new branch] ngimel/no_gc_leakcheck -> origin/ngimel/no_gc_leakcheck 2022-03-17T19:04:59.1160971Z * [new branch] ngimel/norm_backward -> origin/ngimel/norm_backward 2022-03-17T19:04:59.1162892Z * [new branch] ngimel/opinfo_aap -> origin/ngimel/opinfo_aap 2022-03-17T19:04:59.1163912Z * [new branch] ngimel/remove_utilscpp -> origin/ngimel/remove_utilscpp 2022-03-17T19:04:59.1165055Z * [new branch] ngimel/renorm_fix -> origin/ngimel/renorm_fix 2022-03-17T19:04:59.1166285Z * [new branch] ngimel/rnn_context -> origin/ngimel/rnn_context 2022-03-17T19:04:59.1167557Z * [new branch] ngimel/s_where -> origin/ngimel/s_where 2022-03-17T19:04:59.1168900Z * [new branch] ngimel/stack_cat -> origin/ngimel/stack_cat 2022-03-17T19:04:59.1170240Z * [new branch] ngimel/std_var -> origin/ngimel/std_var 2022-03-17T19:04:59.1171672Z * [new branch] ngimel/storage_types -> origin/ngimel/storage_types 2022-03-17T19:04:59.1172872Z * [new branch] ngimel/thc_reduce -> origin/ngimel/thc_reduce 2022-03-17T19:04:59.1174141Z * [new branch] ngimel/unify_norm -> origin/ngimel/unify_norm 2022-03-17T19:04:59.1175418Z * [new branch] ngimel/unique_sliced -> origin/ngimel/unique_sliced 2022-03-17T19:04:59.1176647Z * [new branch] ngimel/view_optim -> origin/ngimel/view_optim 2022-03-17T19:04:59.1177668Z * [new branch] ngimel/zeta_acc -> origin/ngimel/zeta_acc 2022-03-17T19:04:59.1179362Z * [new branch] nightly -> origin/nightly 2022-03-17T19:04:59.1180642Z * [new branch] nightlykickoff -> origin/nightlykickoff 2022-03-17T19:04:59.1182712Z * [new branch] nikitaved/bce_loss_2xbackward_fix -> origin/nikitaved/bce_loss_2xbackward_fix 2022-03-17T19:04:59.1183726Z * [new branch] nikitaved/coo_index_select -> origin/nikitaved/coo_index_select 2022-03-17T19:04:59.1185129Z * [new branch] nikitaved/det_backward -> origin/nikitaved/det_backward 2022-03-17T19:04:59.1186657Z * [new branch] nikitaved/det_double_backward_svd_path -> origin/nikitaved/det_double_backward_svd_path 2022-03-17T19:04:59.1187608Z * [new branch] nikitaved/det_ima_cuda -> origin/nikitaved/det_ima_cuda 2022-03-17T19:04:59.1188900Z * [new branch] nikitaved/det_robust_backward -> origin/nikitaved/det_robust_backward 2022-03-17T19:04:59.1190346Z * [new branch] nikitaved/fix_max_unpool_backward -> origin/nikitaved/fix_max_unpool_backward 2022-03-17T19:04:59.1191543Z * [new branch] nikitaved/householder_forward_ad -> origin/nikitaved/householder_forward_ad 2022-03-17T19:04:59.1193349Z * [new branch] nikitaved/householder_product_optimal_backward -> origin/nikitaved/householder_product_optimal_backward 2022-03-17T19:04:59.1194314Z * [new branch] nikitaved/linalg_qr_forward_AD -> origin/nikitaved/linalg_qr_forward_AD 2022-03-17T19:04:59.1195744Z * [new branch] nikitaved/linalg_svd_rank_revealing -> origin/nikitaved/linalg_svd_rank_revealing 2022-03-17T19:04:59.1196893Z * [new branch] nikitaved/lobpcg_backward_ctx -> origin/nikitaved/lobpcg_backward_ctx 2022-03-17T19:04:59.1198383Z * [new branch] nikitaved/lstsq_forward_backward_ad -> origin/nikitaved/lstsq_forward_backward_ad 2022-03-17T19:04:59.1199425Z * [new branch] nikitaved/lu_backward -> origin/nikitaved/lu_backward 2022-03-17T19:04:59.1200828Z * [new branch] nikitaved/lu_comlex_backward -> origin/nikitaved/lu_comlex_backward 2022-03-17T19:04:59.1202086Z * [new branch] nikitaved/lu_forward_ad -> origin/nikitaved/lu_forward_ad 2022-03-17T19:04:59.1204320Z * [new branch] nikitaved/lu_solve_backward -> origin/nikitaved/lu_solve_backward 2022-03-17T19:04:59.1205440Z * [new branch] nikitaved/lu_solve_forward_ad -> origin/nikitaved/lu_solve_forward_ad 2022-03-17T19:04:59.1206886Z * [new branch] nikitaved/lu_unpack_forward_ad -> origin/nikitaved/lu_unpack_forward_ad 2022-03-17T19:04:59.1208303Z * [new branch] nikitaved/make_bin_cross_entropy_diff_wrt_target -> origin/nikitaved/make_bin_cross_entropy_diff_wrt_target 2022-03-17T19:04:59.1209506Z * [new branch] nikitaved/matrix_rank_missing_interfaces -> origin/nikitaved/matrix_rank_missing_interfaces 2022-03-17T19:04:59.1211014Z * [new branch] nikitaved/max_unpool_opinfos_easier_backward -> origin/nikitaved/max_unpool_opinfos_easier_backward 2022-03-17T19:04:59.1212037Z * [new branch] nikitaved/opinfo_max_pool -> origin/nikitaved/opinfo_max_pool 2022-03-17T19:04:59.1213467Z * [new branch] nikitaved/partial_grad_test -> origin/nikitaved/partial_grad_test 2022-03-17T19:04:59.1214684Z * [new branch] nikitaved/pinv_backward -> origin/nikitaved/pinv_backward 2022-03-17T19:04:59.1216437Z * [new branch] nikitaved/remove_index_fill_side_effect_for_scalar_input -> origin/nikitaved/remove_index_fill_side_effect_for_scalar_input 2022-03-17T19:04:59.1217524Z * [new branch] nikitaved/solve_methods_forward_AD -> origin/nikitaved/solve_methods_forward_AD 2022-03-17T19:04:59.1218719Z * [new branch] nikitaved/stable_sort_CPU -> origin/nikitaved/stable_sort_CPU 2022-03-17T19:04:59.1220177Z * [new branch] nikitaved/stable_sort_CPU_take_2 -> origin/nikitaved/stable_sort_CPU_take_2 2022-03-17T19:04:59.1221352Z * [new branch] nikitaved/svd_lowrank_opinfo -> origin/nikitaved/svd_lowrank_opinfo 2022-03-17T19:04:59.1222786Z * [new branch] nikitaved/test_neg_view_det_singular -> origin/nikitaved/test_neg_view_det_singular 2022-03-17T19:04:59.1224087Z * [new branch] nikitha_language_spec -> origin/nikitha_language_spec 2022-03-17T19:04:59.1225560Z * [new branch] nikitha_stringDedantSupport -> origin/nikitha_stringDedantSupport 2022-03-17T19:04:59.1227287Z * [new branch] nikved/cumsum_complex_backward -> origin/nikved/cumsum_complex_backward 2022-03-17T19:04:59.1228475Z * [new branch] nikved/eig_complex_backward -> origin/nikved/eig_complex_backward 2022-03-17T19:04:59.1229911Z * [new branch] nikved/gather_add -> origin/nikved/gather_add 2022-03-17T19:04:59.1231061Z * [new branch] nikved/index_fill_port -> origin/nikved/index_fill_port 2022-03-17T19:04:59.1232346Z * [new branch] nikved/linalg_lstsq -> origin/nikved/linalg_lstsq 2022-03-17T19:04:59.1233666Z * [new branch] nikved/lobpcg_remove_inverse -> origin/nikved/lobpcg_remove_inverse 2022-03-17T19:04:59.1235208Z * [new branch] nikved/native_lu_unpack -> origin/nikved/native_lu_unpack 2022-03-17T19:04:59.1236288Z * [new branch] nikved/stable_sort_CPU -> origin/nikved/stable_sort_CPU 2022-03-17T19:04:59.1237689Z * [new branch] nll_backward_2 -> origin/nll_backward_2 2022-03-17T19:04:59.1239103Z * [new branch] nnapi_log_softmax -> origin/nnapi_log_softmax 2022-03-17T19:04:59.1240394Z * [new branch] no_zerodim_type_promo -> origin/no_zerodim_type_promo 2022-03-17T19:04:59.1241703Z * [new branch] non_overlapping -> origin/non_overlapping 2022-03-17T19:04:59.1243282Z * [new branch] nonzero_opinfo -> origin/nonzero_opinfo 2022-03-17T19:04:59.1244652Z * [new branch] norm -> origin/norm 2022-03-17T19:04:59.1246011Z * [new branch] norm_docs -> origin/norm_docs 2022-03-17T19:04:59.1247388Z * [new branch] normal_fix -> origin/normal_fix 2022-03-17T19:04:59.1248856Z * [new branch] normalization -> origin/normalization 2022-03-17T19:04:59.1249926Z * [new branch] nt_ops -> origin/nt_ops 2022-03-17T19:04:59.1251452Z * [new branch] nuke -> origin/nuke 2022-03-17T19:04:59.1252989Z * [new branch] on_pull_request_conditional -> origin/on_pull_request_conditional 2022-03-17T19:04:59.1254172Z * [new branch] onnx -> origin/onnx 2022-03-17T19:04:59.1255678Z * [new branch] onnx_fix_sum_keepdims -> origin/onnx_fix_sum_keepdims 2022-03-17T19:04:59.1257087Z * [new branch] onnx_local_function_opset_ms -> origin/onnx_local_function_opset_ms 2022-03-17T19:04:59.1258440Z * [new branch] onnx_ms_1 -> origin/onnx_ms_1 2022-03-17T19:04:59.1259826Z * [new branch] onnx_ms_1_1_22 -> origin/onnx_ms_1_1_22 2022-03-17T19:04:59.1261163Z * [new branch] onnx_quant_bias -> origin/onnx_quant_bias 2022-03-17T19:04:59.1262589Z * [new branch] opaque -> origin/opaque 2022-03-17T19:04:59.1264090Z * [new branch] opaque-dispatch -> origin/opaque-dispatch 2022-03-17T19:04:59.1265734Z * [new branch] opinfo/nll_loss -> origin/opinfo/nll_loss 2022-03-17T19:04:59.1266988Z * [new branch] opinfo/one_hot -> origin/opinfo/one_hot 2022-03-17T19:04:59.1268412Z * [new branch] opinfos_margin_loss -> origin/opinfos_margin_loss 2022-03-17T19:04:59.1269750Z * [new branch] opinfos_pooling -> origin/opinfos_pooling 2022-03-17T19:04:59.1271276Z * [new branch] opinfos_unassigned_misc -> origin/opinfos_unassigned_misc 2022-03-17T19:04:59.1272546Z * [new branch] ops_in_python -> origin/ops_in_python 2022-03-17T19:04:59.1273863Z * [new branch] opti -> origin/opti 2022-03-17T19:04:59.1275277Z * [new branch] optimizers_struct_based -> origin/optimizers_struct_based 2022-03-17T19:04:59.1276621Z * [new branch] optimizers_struct_based_v2 -> origin/optimizers_struct_based_v2 2022-03-17T19:04:59.1277895Z * [new branch] optrm -> origin/optrm 2022-03-17T19:04:59.1280352Z * [new branch] orig/lts/release/1.8 -> origin/orig/lts/release/1.8 2022-03-17T19:04:59.1282077Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2022-03-17T19:04:59.1283647Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2022-03-17T19:04:59.1284764Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2022-03-17T19:04:59.1286040Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2022-03-17T19:04:59.1287575Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2022-03-17T19:04:59.1288523Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2022-03-17T19:04:59.1290295Z * [new branch] origin/fmod-remainder-opinfo -> origin/origin/fmod-remainder-opinfo 2022-03-17T19:04:59.1291230Z * [new branch] origin/record-time -> origin/origin/record-time 2022-03-17T19:04:59.1292744Z * [new branch] paddingsame -> origin/paddingsame 2022-03-17T19:04:59.1294234Z * [new branch] patrickkan/test-help-message -> origin/patrickkan/test-help-message 2022-03-17T19:04:59.1295818Z * [new branch] pavithran/flatbuffer_jit_file -> origin/pavithran/flatbuffer_jit_file 2022-03-17T19:04:59.1297784Z * [new branch] pavithran/flatbuffer_mobile_debug_info -> origin/pavithran/flatbuffer_mobile_debug_info 2022-03-17T19:04:59.1298800Z * [new branch] pbelevich-patch-1 -> origin/pbelevich-patch-1 2022-03-17T19:04:59.1300398Z * [new branch] pbelevich-patch-2 -> origin/pbelevich-patch-2 2022-03-17T19:04:59.1301651Z * [new branch] pbelevich-patch-3 -> origin/pbelevich-patch-3 2022-03-17T19:04:59.1303157Z * [new branch] pdt_tutorial -> origin/pdt_tutorial 2022-03-17T19:04:59.1304534Z * [new branch] per_channel_quant -> origin/per_channel_quant 2022-03-17T19:04:59.1305731Z * [new branch] peterjc123-patch-1 -> origin/peterjc123-patch-1 2022-03-17T19:04:59.1307290Z * [new branch] peterjc123-patch-1-1 -> origin/peterjc123-patch-1-1 2022-03-17T19:04:59.1308356Z * [new branch] pg_fix -> origin/pg_fix 2022-03-17T19:04:59.1310090Z * [new branch] pickle-doc-warning -> origin/pickle-doc-warning 2022-03-17T19:04:59.1311544Z * [new branch] poc_remove_sth_from_torch_nn_module -> origin/poc_remove_sth_from_torch_nn_module 2022-03-17T19:04:59.1312645Z * [new branch] portforward_fix -> origin/portforward_fix 2022-03-17T19:04:59.1319489Z * [new branch] postnightly -> origin/postnightly 2022-03-17T19:04:59.1319948Z * [new branch] pr-33000 -> origin/pr-33000 2022-03-17T19:04:59.1320455Z * [new branch] primTorch_aliases -> origin/primTorch_aliases 2022-03-17T19:04:59.1321019Z * [new branch] pritam/dist_autograd_sync -> origin/pritam/dist_autograd_sync 2022-03-17T19:04:59.1321558Z * [new branch] process_commit_update2 -> origin/process_commit_update2 2022-03-17T19:04:59.1322208Z * [new branch] profile -> origin/profile 2022-03-17T19:04:59.1323103Z * [new branch] profiler_docs -> origin/profiler_docs 2022-03-17T19:04:59.1324250Z * [new branch] profiler_fallback -> origin/profiler_fallback 2022-03-17T19:04:59.1325722Z * [new branch] proto_jiterated_complex -> origin/proto_jiterated_complex 2022-03-17T19:04:59.1326827Z * [new branch] protoissue -> origin/protoissue 2022-03-17T19:04:59.1328383Z * [new branch] prototype_proc -> origin/prototype_proc 2022-03-17T19:04:59.1329724Z * [new branch] proxy_tensor -> origin/proxy_tensor 2022-03-17T19:04:59.1331135Z * [new branch] pt_mobile_api_fc_tmp -> origin/pt_mobile_api_fc_tmp 2022-03-17T19:04:59.1333020Z * [new branch] pull/24055/head -> origin/pull/24055/head 2022-03-17T19:04:59.1334726Z * [new branch] python391-annotation_str -> origin/python391-annotation_str 2022-03-17T19:04:59.1336051Z * [new branch] python_mode -> origin/python_mode 2022-03-17T19:04:59.1337344Z * [new branch] python_mode2 -> origin/python_mode2 2022-03-17T19:04:59.1338907Z * [new branch] pytorch-test_xla_ci2 -> origin/pytorch-test_xla_ci2 2022-03-17T19:04:59.1340397Z * [new branch] pytree_fx -> origin/pytree_fx 2022-03-17T19:04:59.1341546Z * [new branch] qnnpack_dynamic -> origin/qnnpack_dynamic 2022-03-17T19:04:59.1343051Z * [new branch] qnnpack_per_channel_quant -> origin/qnnpack_per_channel_quant 2022-03-17T19:04:59.1344521Z * [new branch] quantization_1_3_doc -> origin/quantization_1_3_doc 2022-03-17T19:04:59.1345667Z * [new branch] quantized_hardtanh -> origin/quantized_hardtanh 2022-03-17T19:04:59.1347065Z * [new branch] quick-fix -> origin/quick-fix 2022-03-17T19:04:59.1348429Z * [new branch] random_test -> origin/random_test 2022-03-17T19:04:59.1349803Z * [new branch] random_testing -> origin/random_testing 2022-03-17T19:04:59.1351337Z * [new branch] re-enable-mac-json-uploads -> origin/re-enable-mac-json-uploads 2022-03-17T19:04:59.1352648Z * [new branch] recursive_module_deepcopy -> origin/recursive_module_deepcopy 2022-03-17T19:04:59.1354086Z * [new branch] redo-override-listing -> origin/redo-override-listing 2022-03-17T19:04:59.1355476Z * [new branch] reduction_type_promotion -> origin/reduction_type_promotion 2022-03-17T19:04:59.1356826Z * [new branch] refactor_build_wheel -> origin/refactor_build_wheel 2022-03-17T19:04:59.1358146Z * [new branch] reference_inputs -> origin/reference_inputs 2022-03-17T19:04:59.1359935Z * [new branch] release/1.10 -> origin/release/1.10 2022-03-17T19:04:59.1361293Z * [new branch] release/1.11 -> origin/release/1.11 2022-03-17T19:04:59.1362749Z * [new branch] release/1.11_a -> origin/release/1.11_a 2022-03-17T19:04:59.1364036Z * [new branch] release/1.4 -> origin/release/1.4 2022-03-17T19:04:59.1365084Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2022-03-17T19:04:59.1366553Z * [new branch] release/1.5 -> origin/release/1.5 2022-03-17T19:04:59.1367856Z * [new branch] release/1.6 -> origin/release/1.6 2022-03-17T19:04:59.1369219Z * [new branch] release/1.7 -> origin/release/1.7 2022-03-17T19:04:59.1370267Z * [new branch] release/1.8 -> origin/release/1.8 2022-03-17T19:04:59.1371718Z * [new branch] release/1.9 -> origin/release/1.9 2022-03-17T19:04:59.1373269Z * [new branch] release_1.8_fq_docs_patch -> origin/release_1.8_fq_docs_patch 2022-03-17T19:04:59.1374621Z * [new branch] remove-root-job -> origin/remove-root-job 2022-03-17T19:04:59.1376109Z * [new branch] remove-test-history-script -> origin/remove-test-history-script 2022-03-17T19:04:59.1377618Z * [new branch] remove-win-smoke-tests -> origin/remove-win-smoke-tests 2022-03-17T19:04:59.1378693Z * [new branch] removeAnnoyingWarnings -> origin/removeAnnoyingWarnings 2022-03-17T19:04:59.1380216Z * [new branch] remove_duplicate -> origin/remove_duplicate 2022-03-17T19:04:59.1381646Z * [new branch] remove_legacy_definitions -> origin/remove_legacy_definitions 2022-03-17T19:04:59.1382820Z * [new branch] remove_opinfo_warning -> origin/remove_opinfo_warning 2022-03-17T19:04:59.1384228Z * [new branch] remove_prints -> origin/remove_prints 2022-03-17T19:04:59.1385646Z * [new branch] remove_redundant_print -> origin/remove_redundant_print 2022-03-17T19:04:59.1386955Z * [new branch] remove_test -> origin/remove_test 2022-03-17T19:04:59.1388437Z * [new branch] rename-quant-module -> origin/rename-quant-module 2022-03-17T19:04:59.1390101Z * [new branch] rename-quant-module2 -> origin/rename-quant-module2 2022-03-17T19:04:59.1391890Z * [new branch] rename_index_to_advanced_index -> origin/rename_index_to_advanced_index 2022-03-17T19:04:59.1393055Z * [new branch] resnet_forward_ad_ops -> origin/resnet_forward_ad_ops 2022-03-17T19:04:59.1395350Z * [new branch] revert-55729-gh/msaroufim/5/head -> origin/revert-55729-gh/msaroufim/5/head 2022-03-17T19:04:59.1397228Z * [new branch] revert-62257-deyu/remove_use_external_data_format -> origin/revert-62257-deyu/remove_use_external_data_format 2022-03-17T19:04:59.1399071Z * [new branch] revert-62766-gh/BowenBao/109/head -> origin/revert-62766-gh/BowenBao/109/head 2022-03-17T19:04:59.1401301Z * [new branch] revert-64051-gh/jaceyca/2/head -> origin/revert-64051-gh/jaceyca/2/head 2022-03-17T19:04:59.1405761Z * [new branch] revert-65014-gh/mcr229/1/head -> origin/revert-65014-gh/mcr229/1/head 2022-03-17T19:04:59.1407784Z * [new branch] revert-66735-gh/PaliC/5/head -> origin/revert-66735-gh/PaliC/5/head 2022-03-17T19:04:59.1409997Z * [new branch] revert-69916-gh/wanchaol/193/head -> origin/revert-69916-gh/wanchaol/193/head 2022-03-17T19:04:59.1412354Z * [new branch] revert-69943-revert-69916-gh/wanchaol/193/head -> origin/revert-69943-revert-69916-gh/wanchaol/193/head 2022-03-17T19:04:59.1414356Z * [new branch] revert-73214-gh/H-Huang/47/head -> origin/revert-73214-gh/H-Huang/47/head 2022-03-17T19:04:59.1415428Z * [new branch] revivify_ci -> origin/revivify_ci 2022-03-17T19:04:59.1416939Z * [new branch] rnn_reduce_range -> origin/rnn_reduce_range 2022-03-17T19:04:59.1418604Z * [new branch] robieta/debug_info -> origin/robieta/debug_info 2022-03-17T19:04:59.1419759Z * [new branch] robieta/module_tracking -> origin/robieta/module_tracking 2022-03-17T19:04:59.1421288Z * [new branch] robieta/module_tracking_reland -> origin/robieta/module_tracking_reland 2022-03-17T19:04:59.1422344Z * [new branch] robieta/timer_div_zero -> origin/robieta/timer_div_zero 2022-03-17T19:04:59.1423823Z * [new branch] robieta/torchtidy_prototype -> origin/robieta/torchtidy_prototype 2022-03-17T19:04:59.1425320Z * [new branch] rocm-dist-master -> origin/rocm-dist-master 2022-03-17T19:04:59.1426380Z * [new branch] rocm_gha -> origin/rocm_gha 2022-03-17T19:04:59.1427717Z * [new branch] rocm_slow_test_label -> origin/rocm_slow_test_label 2022-03-17T19:04:59.1429786Z * [new branch] rohan-varma-patch-1 -> origin/rohan-varma-patch-1 2022-03-17T19:04:59.1430938Z * [new branch] rohan-varma-patch-2 -> origin/rohan-varma-patch-2 2022-03-17T19:04:59.1432148Z * [new branch] rohan_fix -> origin/rohan_fix 2022-03-17T19:04:59.1433697Z * [new branch] rpc_dev -> origin/rpc_dev 2022-03-17T19:04:59.1435229Z * [new branch] running-generate_ci_workflows -> origin/running-generate_ci_workflows 2022-03-17T19:04:59.1436426Z * [new branch] saketh-modules-lab -> origin/saketh-modules-lab 2022-03-17T19:04:59.1438225Z * [new branch] samestep/docker-cache-gha -> origin/samestep/docker-cache-gha 2022-03-17T19:04:59.1439634Z * [new branch] samestep/gha-ossci-metrics-new-token -> origin/samestep/gha-ossci-metrics-new-token 2022-03-17T19:04:59.1440863Z * [new branch] samestep/redundant-workflows-base -> origin/samestep/redundant-workflows-base 2022-03-17T19:04:59.1442230Z * [new branch] saved-tensors-docstring -> origin/saved-tensors-docstring 2022-03-17T19:04:59.1443501Z * [new branch] saved_tensors_docs -> origin/saved_tensors_docs 2022-03-17T19:04:59.1444805Z * [new branch] scalarListAPI -> origin/scalarListAPI 2022-03-17T19:04:59.1446528Z * [new branch] scatter-reductions-cpu -> origin/scatter-reductions-cpu 2022-03-17T19:04:59.1447715Z * [new branch] scatter_gather_more_shape_checks -> origin/scatter_gather_more_shape_checks 2022-03-17T19:04:59.1448931Z * [new branch] scatter_gather_to_ATen_CUDA -> origin/scatter_gather_to_ATen_CUDA 2022-03-17T19:04:59.1450803Z * [new branch] sdym/check-default-branch -> origin/sdym/check-default-branch 2022-03-17T19:04:59.1451889Z * [new branch] sdym/fix-54979 -> origin/sdym/fix-54979 2022-03-17T19:04:59.1453219Z * [new branch] sdym/fix-undefined_internal -> origin/sdym/fix-undefined_internal 2022-03-17T19:04:59.1454553Z * [new branch] sdym/generated-stats -> origin/sdym/generated-stats 2022-03-17T19:04:59.1455855Z * [new branch] sdym/gha-android-test -> origin/sdym/gha-android-test 2022-03-17T19:04:59.1457205Z * [new branch] sdym/py310-test_binary_ufuncs.py -> origin/sdym/py310-test_binary_ufuncs.py 2022-03-17T19:04:59.1458526Z * [new branch] sdym/py310-test_distributions -> origin/sdym/py310-test_distributions 2022-03-17T19:04:59.1459684Z * [new branch] sdym/qinterpolate -> origin/sdym/qinterpolate 2022-03-17T19:04:59.1460993Z * [new branch] sdym/test-merge -> origin/sdym/test-merge 2022-03-17T19:04:59.1462293Z * [new branch] sdym/test-mobile-rules -> origin/sdym/test-mobile-rules 2022-03-17T19:04:59.1463698Z * [new branch] seemethere-patch-1 -> origin/seemethere-patch-1 2022-03-17T19:04:59.1465547Z * [new branch] seemethere/actions_wheels -> origin/seemethere/actions_wheels 2022-03-17T19:04:59.1466787Z * [new branch] seemethere/add_core_to_ghf -> origin/seemethere/add_core_to_ghf 2022-03-17T19:04:59.1468080Z * [new branch] seemethere/fix_collect_env -> origin/seemethere/fix_collect_env 2022-03-17T19:04:59.1469695Z * [new branch] seemethere/formalize-patch-release-process -> origin/seemethere/formalize-patch-release-process 2022-03-17T19:04:59.1470693Z * [new branch] seemethere/nightly_pls_work -> origin/seemethere/nightly_pls_work 2022-03-17T19:04:59.1472093Z * [new branch] seemethere/pull_request_target -> origin/seemethere/pull_request_target 2022-03-17T19:04:59.1473405Z * [new branch] seemethere/rebuild_rocm_stable_image -> origin/seemethere/rebuild_rocm_stable_image 2022-03-17T19:04:59.1474972Z * [new branch] seemethere/rebuilding_lts_18_docker_images -> origin/seemethere/rebuilding_lts_18_docker_images 2022-03-17T19:04:59.1476205Z * [new branch] seemethere/refactor_build_wheel -> origin/seemethere/refactor_build_wheel 2022-03-17T19:04:59.1477489Z * [new branch] seemethere/rocm_not_on_master -> origin/seemethere/rocm_not_on_master 2022-03-17T19:04:59.1478767Z * [new branch] seemethere/test_collect_env -> origin/seemethere/test_collect_env 2022-03-17T19:04:59.1480189Z * [new branch] seemethere/update_branch_cut -> origin/seemethere/update_branch_cut 2022-03-17T19:04:59.1481782Z * [new branch] serialize_undefined_tensor -> origin/serialize_undefined_tensor 2022-03-17T19:04:59.1483212Z * [new branch] set_default_dtype_docs -> origin/set_default_dtype_docs 2022-03-17T19:04:59.1484393Z * [new branch] shape_info -> origin/shape_info 2022-03-17T19:04:59.1485760Z * [new branch] shen-fixes -> origin/shen-fixes 2022-03-17T19:04:59.1487242Z * [new branch] side_stream_patch -> origin/side_stream_patch 2022-03-17T19:04:59.1488570Z * [new branch] silent-rm-error -> origin/silent-rm-error 2022-03-17T19:04:59.1490006Z * [new branch] size_investigation -> origin/size_investigation 2022-03-17T19:04:59.1491424Z * [new branch] skip_meta_conversion -> origin/skip_meta_conversion 2022-03-17T19:04:59.1492810Z * [new branch] sln -> origin/sln 2022-03-17T19:04:59.1494395Z * [new branch] smessmer/lto -> origin/smessmer/lto 2022-03-17T19:04:59.1495642Z * [new branch] smessmer/scalar -> origin/smessmer/scalar 2022-03-17T19:04:59.1496958Z * [new branch] sort-cxx11 -> origin/sort-cxx11 2022-03-17T19:04:59.1498414Z * [new branch] sort-segment -> origin/sort-segment 2022-03-17T19:04:59.1499922Z * [new branch] soulitzer-patch-1 -> origin/soulitzer-patch-1 2022-03-17T19:04:59.1501835Z * [new branch] soulitzer/benchmarking -> origin/soulitzer/benchmarking 2022-03-17T19:04:59.1502924Z * [new branch] source_range -> origin/source_range 2022-03-17T19:04:59.1504289Z * [new branch] sparse-csr-layout -> origin/sparse-csr-layout 2022-03-17T19:04:59.1505651Z * [new branch] sparse-error-msgs -> origin/sparse-error-msgs 2022-03-17T19:04:59.1507052Z * [new branch] sparse_rpc_fix -> origin/sparse_rpc_fix 2022-03-17T19:04:59.1508339Z * [new branch] stack2 -> origin/stack2 2022-03-17T19:04:59.1509674Z * [new branch] stale -> origin/stale 2022-03-17T19:04:59.1511111Z * [new branch] static_dispatch -> origin/static_dispatch 2022-03-17T19:04:59.1512328Z * [new branch] stream_patch -> origin/stream_patch 2022-03-17T19:04:59.1513698Z * [new branch] streamcontext_nop -> origin/streamcontext_nop 2022-03-17T19:04:59.1514585Z * [new branch] stringDedentSupport -> origin/stringDedentSupport 2022-03-17T19:04:59.1515931Z * [new branch] string_view -> origin/string_view 2022-03-17T19:04:59.1517396Z * [new branch] structured_amax -> origin/structured_amax 2022-03-17T19:04:59.1518761Z * [new branch] structured_mse_loss -> origin/structured_mse_loss 2022-03-17T19:04:59.1520092Z * [new branch] su_ghf -> origin/su_ghf 2022-03-17T19:04:59.1521509Z * [new branch] sub_iter_strides_cuda -> origin/sub_iter_strides_cuda 2022-03-17T19:04:59.1523930Z * [new branch] suo-patch-1 -> origin/suo-patch-1 2022-03-17T19:04:59.1525239Z * [new branch] suo-patch-2 -> origin/suo-patch-2 2022-03-17T19:04:59.1526647Z * [new branch] suo-patch-8 -> origin/suo-patch-8 2022-03-17T19:04:59.1527979Z * [new branch] sup -> origin/sup 2022-03-17T19:04:59.1529723Z * [new branch] support_any -> origin/support_any 2022-03-17T19:04:59.1531113Z * [new branch] support_complex -> origin/support_complex 2022-03-17T19:04:59.1532505Z * [new branch] support_pow -> origin/support_pow 2022-03-17T19:04:59.1533841Z * [new branch] support_sum -> origin/support_sum 2022-03-17T19:04:59.1535357Z * [new branch] support_type_inference_nn_modules -> origin/support_type_inference_nn_modules 2022-03-17T19:04:59.1536690Z * [new branch] suraj813-patch-1 -> origin/suraj813-patch-1 2022-03-17T19:04:59.1538085Z * [new branch] svdvals -> origin/svdvals 2022-03-17T19:04:59.1539506Z * [new branch] syncbatchnorm-0-batch-dim -> origin/syncbatchnorm-0-batch-dim 2022-03-17T19:04:59.1540821Z * [new branch] t-opinfo -> origin/t-opinfo 2022-03-17T19:04:59.1542295Z * [new branch] templated-boxed-fallback -> origin/templated-boxed-fallback 2022-03-17T19:04:59.1543659Z * [new branch] tensor-iterator-manual -> origin/tensor-iterator-manual 2022-03-17T19:04:59.1545057Z * [new branch] tensor_docs -> origin/tensor_docs 2022-03-17T19:04:59.1546316Z * [new branch] tensor_proto -> origin/tensor_proto 2022-03-17T19:04:59.1548103Z * [new branch] tensordot_bug -> origin/tensordot_bug 2022-03-17T19:04:59.1549122Z * [new branch] test-master -> origin/test-master 2022-03-17T19:04:59.1550702Z * [new branch] test-parse-ref-on-win -> origin/test-parse-ref-on-win 2022-03-17T19:04:59.1551987Z * [new branch] test_35736 -> origin/test_35736 2022-03-17T19:04:59.1553429Z * [new branch] test_deploy -> origin/test_deploy 2022-03-17T19:04:59.1554835Z * [new branch] test_equiv_recurrent_disable -> origin/test_equiv_recurrent_disable 2022-03-17T19:04:59.1555836Z * [new branch] test_flaky -> origin/test_flaky 2022-03-17T19:04:59.1557494Z * [new branch] test_functional_non_const_ref -> origin/test_functional_non_const_ref 2022-03-17T19:04:59.1558647Z * [new branch] test_k_0312 -> origin/test_k_0312 2022-03-17T19:04:59.1560121Z * [new branch] test_no_warns -> origin/test_no_warns 2022-03-17T19:04:59.1561622Z * [new branch] test_ops_to_support_iterables -> origin/test_ops_to_support_iterables 2022-03-17T19:04:59.1562919Z * [new branch] test_pr -> origin/test_pr 2022-03-17T19:04:59.1564489Z * [new branch] test_rocm_35959 -> origin/test_rocm_35959 2022-03-17T19:04:59.1565851Z * [new branch] test_xla_codegen_changes -> origin/test_xla_codegen_changes 2022-03-17T19:04:59.1567186Z * [new branch] testing_lint_fix -> origin/testing_lint_fix 2022-03-17T19:04:59.1568450Z * [new branch] tf32 -> origin/tf32 2022-03-17T19:04:59.1570021Z * [new branch] thread-local-default-hooks -> origin/thread-local-default-hooks 2022-03-17T19:04:59.1571218Z * [new branch] tktrungna-patch-1 -> origin/tktrungna-patch-1 2022-03-17T19:04:59.1572673Z * [new branch] tmp-codeowner -> origin/tmp-codeowner 2022-03-17T19:04:59.1574026Z * [new branch] torch_version_update -> origin/torch_version_update 2022-03-17T19:04:59.1575378Z * [new branch] trace_fn_calls2 -> origin/trace_fn_calls2 2022-03-17T19:04:59.1576826Z * [new branch] triangular_solve -> origin/triangular_solve 2022-03-17T19:04:59.1578356Z * [new branch] triangular_solve_permute -> origin/triangular_solve_permute 2022-03-17T19:04:59.1579512Z * [new branch] try-hacking-unique -> origin/try-hacking-unique 2022-03-17T19:04:59.1580998Z * [new branch] try-retryable-steps -> origin/try-retryable-steps 2022-03-17T19:04:59.1582257Z * [new branch] tt_emb -> origin/tt_emb 2022-03-17T19:04:59.1583675Z * [new branch] turn_on_cpp_api_parity_test -> origin/turn_on_cpp_api_parity_test 2022-03-17T19:04:59.1585114Z * [new branch] turn_on_cpp_api_parity_test_new_new -> origin/turn_on_cpp_api_parity_test_new_new 2022-03-17T19:04:59.1586746Z * [new branch] turn_on_cpp_api_parity_test_v2 -> origin/turn_on_cpp_api_parity_test_v2 2022-03-17T19:04:59.1587981Z * [new branch] turn_on_cpp_api_parity_test_v3 -> origin/turn_on_cpp_api_parity_test_v3 2022-03-17T19:04:59.1589450Z * [new branch] turn_on_cpp_api_parity_test_v3_exp1 -> origin/turn_on_cpp_api_parity_test_v3_exp1 2022-03-17T19:04:59.1590683Z * [new branch] turn_on_cpp_api_parity_test_v3_exp2 -> origin/turn_on_cpp_api_parity_test_v3_exp2 2022-03-17T19:04:59.1591991Z * [new branch] type_system_1 -> origin/type_system_1 2022-03-17T19:04:59.1593276Z * [new branch] ucc -> origin/ucc 2022-03-17T19:04:59.1594841Z * [new branch] ucc-build -> origin/ucc-build 2022-03-17T19:04:59.1596238Z * [new branch] ucc-test- -> origin/ucc-test- 2022-03-17T19:04:59.1597316Z * [new branch] unbreak_bc -> origin/unbreak_bc 2022-03-17T19:04:59.1599214Z * [new branch] unique_opinfo -> origin/unique_opinfo 2022-03-17T19:04:59.1600258Z * [new branch] upd_k_0609 -> origin/upd_k_0609 2022-03-17T19:04:59.1601773Z * [new branch] upd_kineto_0324 -> origin/upd_kineto_0324 2022-03-17T19:04:59.1603344Z * [new branch] upd_kineto_0330 -> origin/upd_kineto_0330 2022-03-17T19:04:59.1605030Z * [new branch] upd_kineto_0804 -> origin/upd_kineto_0804 2022-03-17T19:04:59.1606384Z * [new branch] upd_kineto_rev_4 -> origin/upd_kineto_rev_4 2022-03-17T19:04:59.1607717Z * [new branch] update-build-bin -> origin/update-build-bin 2022-03-17T19:04:59.1609063Z * [new branch] update-stale-1 -> origin/update-stale-1 2022-03-17T19:04:59.1610477Z * [new branch] update-test-distribute -> origin/update-test-distribute 2022-03-17T19:04:59.1611843Z * [new branch] update-test-libtorch -> origin/update-test-libtorch 2022-03-17T19:04:59.1613352Z * [new branch] update-test-libtorch-path -> origin/update-test-libtorch-path 2022-03-17T19:04:59.1614426Z * [new branch] update-test-path -> origin/update-test-path 2022-03-17T19:04:59.1615954Z * [new branch] update-test-vulkan-path -> origin/update-test-vulkan-path 2022-03-17T19:04:59.1617345Z * [new branch] update_gloo_04152021 -> origin/update_gloo_04152021 2022-03-17T19:04:59.1618762Z * [new branch] update_gloo_submodule -> origin/update_gloo_submodule 2022-03-17T19:04:59.1619873Z * [new branch] update_kineto_0505 -> origin/update_kineto_0505 2022-03-17T19:04:59.1621533Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2022-03-17T19:04:59.1623206Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2022-03-17T19:04:59.1624751Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2022-03-17T19:04:59.1626731Z * [new branch] upstream/ci-all/record-time-slowtests-krshrimali -> origin/upstream/ci-all/record-time-slowtests-krshrimali 2022-03-17T19:04:59.1627779Z * [new branch] upstream/nikved/linalg_lstsq -> origin/upstream/nikved/linalg_lstsq 2022-03-17T19:04:59.1629159Z * [new branch] v0.1.2 -> origin/v0.1.2 2022-03-17T19:04:59.1630613Z * [new branch] v1.0.1 -> origin/v1.0.1 2022-03-17T19:04:59.1631976Z * [new branch] v1.0.3 -> origin/v1.0.3 2022-03-17T19:04:59.1633275Z * [new branch] v1.1.0 -> origin/v1.1.0 2022-03-17T19:04:59.1634634Z * [new branch] v1.2.0 -> origin/v1.2.0 2022-03-17T19:04:59.1636011Z * [new branch] v1.3.0 -> origin/v1.3.0 2022-03-17T19:04:59.1637376Z * [new branch] v1.3.1 -> origin/v1.3.1 2022-03-17T19:04:59.1638725Z * [new branch] v1.4 -> origin/v1.4 2022-03-17T19:04:59.1640534Z * [new branch] version_mismatch_error_msg -> origin/version_mismatch_error_msg 2022-03-17T19:04:59.1642536Z * [new branch] vfdev-5/docs-typo-fix -> origin/vfdev-5/docs-typo-fix 2022-03-17T19:04:59.1643900Z * [new branch] viab -> origin/viab 2022-03-17T19:04:59.1645578Z * [new branch] viable/strict -> origin/viable/strict 2022-03-17T19:04:59.1646889Z * [new branch] view_as_real -> origin/view_as_real 2022-03-17T19:04:59.1648484Z * [new branch] vision_maskrcnn -> origin/vision_maskrcnn 2022-03-17T19:04:59.1650183Z * [new branch] voznesenskym/named_buffers -> origin/voznesenskym/named_buffers 2022-03-17T19:04:59.1651363Z * [new branch] voznesenskym/named_parameters -> origin/voznesenskym/named_parameters 2022-03-17T19:04:59.1652920Z * [new branch] vs-show-message-on-exception -> origin/vs-show-message-on-exception 2022-03-17T19:04:59.1654183Z * [new branch] vsb_cache_aware_benchmark_preview -> origin/vsb_cache_aware_benchmark_preview 2022-03-17T19:04:59.1655789Z * [new branch] vulkan_add_hardshrink_op -> origin/vulkan_add_hardshrink_op 2022-03-17T19:04:59.1656965Z * [new branch] vulkan_conv2d_old_prepack -> origin/vulkan_conv2d_old_prepack 2022-03-17T19:04:59.1658426Z * [new branch] vulkan_conv_cpushader -> origin/vulkan_conv_cpushader 2022-03-17T19:04:59.1659764Z * [new branch] vulkan_frcnn_ops -> origin/vulkan_frcnn_ops 2022-03-17T19:04:59.1661189Z * [new branch] walterddr-bc-checker-hotfix -> origin/walterddr-bc-checker-hotfix 2022-03-17T19:04:59.1662918Z * [new branch] wconstab/backend -> origin/wconstab/backend 2022-03-17T19:04:59.1664053Z * [new branch] wconstab/bench -> origin/wconstab/bench 2022-03-17T19:04:59.1665397Z * [new branch] wconstab/bench2 -> origin/wconstab/bench2 2022-03-17T19:04:59.1666849Z * [new branch] wconstab/benchit -> origin/wconstab/benchit 2022-03-17T19:04:59.1667834Z * [new branch] wconstab/benchmark -> origin/wconstab/benchmark 2022-03-17T19:04:59.1669398Z * [new branch] wconstab/const_containerhash -> origin/wconstab/const_containerhash 2022-03-17T19:04:59.1670413Z * [new branch] wconstab/dbg_nll -> origin/wconstab/dbg_nll 2022-03-17T19:04:59.1671943Z * [new branch] wconstab/dcgan_ops -> origin/wconstab/dcgan_ops 2022-03-17T19:04:59.1673080Z * [new branch] wconstab/debug -> origin/wconstab/debug 2022-03-17T19:04:59.1674501Z * [new branch] wconstab/devdata -> origin/wconstab/devdata 2022-03-17T19:04:59.1675876Z * [new branch] wconstab/example.py_ci -> origin/wconstab/example.py_ci 2022-03-17T19:04:59.1676919Z * [new branch] wconstab/fallback -> origin/wconstab/fallback 2022-03-17T19:04:59.1678231Z * [new branch] wconstab/fallback2 -> origin/wconstab/fallback2 2022-03-17T19:04:59.1680096Z * [new branch] wconstab/fields -> origin/wconstab/fields 2022-03-17T19:04:59.1681335Z * [new branch] wconstab/fix_clone -> origin/wconstab/fix_clone 2022-03-17T19:04:59.1682711Z * [new branch] wconstab/hash_perf -> origin/wconstab/hash_perf 2022-03-17T19:04:59.1684217Z * [new branch] wconstab/improve_deploy_cpy_build -> origin/wconstab/improve_deploy_cpy_build 2022-03-17T19:04:59.1685290Z * [new branch] wconstab/incorrect -> origin/wconstab/incorrect 2022-03-17T19:04:59.1686706Z * [new branch] wconstab/local_scalar_dense -> origin/wconstab/local_scalar_dense 2022-03-17T19:04:59.1687962Z * [new branch] wconstab/lr_hack -> origin/wconstab/lr_hack 2022-03-17T19:04:59.1689258Z * [new branch] wconstab/ltc-noopt -> origin/wconstab/ltc-noopt 2022-03-17T19:04:59.1690679Z * [new branch] wconstab/ltc-nvfuser-nofallback -> origin/wconstab/ltc-nvfuser-nofallback 2022-03-17T19:04:59.1691959Z * [new branch] wconstab/ltc_codegen -> origin/wconstab/ltc_codegen 2022-03-17T19:04:59.1693359Z * [new branch] wconstab/ltc_codegen_clone -> origin/wconstab/ltc_codegen_clone 2022-03-17T19:04:59.1694653Z * [new branch] wconstab/ltc_core_hash -> origin/wconstab/ltc_core_hash 2022-03-17T19:04:59.1696286Z * [new branch] wconstab/move_ltc_in_tree -> origin/wconstab/move_ltc_in_tree 2022-03-17T19:04:59.1697301Z * [new branch] wconstab/mul -> origin/wconstab/mul 2022-03-17T19:04:59.1698650Z * [new branch] wconstab/native_batch_norm -> origin/wconstab/native_batch_norm 2022-03-17T19:04:59.1699901Z * [new branch] wconstab/node_base -> origin/wconstab/node_base 2022-03-17T19:04:59.1701216Z * [new branch] wconstab/perf -> origin/wconstab/perf 2022-03-17T19:04:59.1702588Z * [new branch] wconstab/refactor_benchmark -> origin/wconstab/refactor_benchmark 2022-03-17T19:04:59.1703799Z * [new branch] wconstab/registrar -> origin/wconstab/registrar 2022-03-17T19:04:59.1705079Z * [new branch] wconstab/remove_hash -> origin/wconstab/remove_hash 2022-03-17T19:04:59.1706361Z * [new branch] wconstab/run_tb -> origin/wconstab/run_tb 2022-03-17T19:04:59.1707703Z * [new branch] wconstab/scalar -> origin/wconstab/scalar 2022-03-17T19:04:59.1709069Z * [new branch] wconstab/shape_cache -> origin/wconstab/shape_cache 2022-03-17T19:04:59.1710399Z * [new branch] wconstab/shape_hash -> origin/wconstab/shape_hash 2022-03-17T19:04:59.1711687Z * [new branch] wconstab/shapeinfr -> origin/wconstab/shapeinfr 2022-03-17T19:04:59.1713231Z * [new branch] wconstab/structured_native_batch_norm -> origin/wconstab/structured_native_batch_norm 2022-03-17T19:04:59.1714396Z * [new branch] wconstab/torch_python_objs -> origin/wconstab/torch_python_objs 2022-03-17T19:04:59.1715611Z * [new branch] wconstab/torchpy -> origin/wconstab/torchpy 2022-03-17T19:04:59.1716907Z * [new branch] win_clang -> origin/win_clang 2022-03-17T19:04:59.1718495Z * [new branch] windows_distributions_tests -> origin/windows_distributions_tests 2022-03-17T19:04:59.1719811Z * [new branch] workflow-per-job -> origin/workflow-per-job 2022-03-17T19:04:59.1721649Z * [new branch] xz9/add-release-testing -> origin/xz9/add-release-testing 2022-03-17T19:04:59.1723231Z * [new branch] xz9/fix-tb-branch-run -> origin/xz9/fix-tb-branch-run 2022-03-17T19:04:59.1724217Z * [new branch] xz9/fix-tb-ci -> origin/xz9/fix-tb-ci 2022-03-17T19:04:59.1725612Z * [new branch] xz9/fix-torchbench-ci-update -> origin/xz9/fix-torchbench-ci-update 2022-03-17T19:04:59.1726850Z * [new branch] xz9/test-ghi-68086 -> origin/xz9/test-ghi-68086 2022-03-17T19:04:59.1728239Z * [new branch] xz9/test-pr-71710 -> origin/xz9/test-pr-71710 2022-03-17T19:04:59.1729474Z * [new branch] xz9/torchbench-test -> origin/xz9/torchbench-test 2022-03-17T19:04:59.1730781Z * [new branch] xz9/verify_ngimel_layer_norm -> origin/xz9/verify_ngimel_layer_norm 2022-03-17T19:04:59.1732049Z * [new branch] xz9/verify_ngimel_layer_norm_2 -> origin/xz9/verify_ngimel_layer_norm_2 2022-03-17T19:04:59.1733760Z * [new branch] xzhao9/update-docker-build -> origin/xzhao9/update-docker-build 2022-03-17T19:04:59.1735028Z * [new branch] yinsun-41101 -> origin/yinsun-41101 2022-03-17T19:04:59.1736465Z * [new branch] zasdfgbnm-patch-4 -> origin/zasdfgbnm-patch-4 2022-03-17T19:04:59.1738192Z * [new branch] zasdfgbnm/gelu -> origin/zasdfgbnm/gelu 2022-03-17T19:04:59.1739614Z * [new branch] zhouzhuojie-patch-1 -> origin/zhouzhuojie-patch-1 2022-03-17T19:04:59.1741273Z * [new branch] zz/add-github.com-noproxy -> origin/zz/add-github.com-noproxy 2022-03-17T19:04:59.1742479Z * [new tag] ciflow/all/63212 -> ciflow/all/63212 2022-03-17T19:04:59.1743206Z * [new tag] ciflow/all/64387 -> ciflow/all/64387 2022-03-17T19:04:59.1744234Z * [new tag] ciflow/all/67022 -> ciflow/all/67022 2022-03-17T19:04:59.1744964Z * [new tag] ciflow/all/67023 -> ciflow/all/67023 2022-03-17T19:04:59.1745676Z * [new tag] ciflow/all/67079 -> ciflow/all/67079 2022-03-17T19:04:59.1746404Z * [new tag] ciflow/all/67343 -> ciflow/all/67343 2022-03-17T19:04:59.1747091Z * [new tag] ciflow/all/67833 -> ciflow/all/67833 2022-03-17T19:04:59.1747777Z * [new tag] ciflow/all/68328 -> ciflow/all/68328 2022-03-17T19:04:59.1748482Z * [new tag] ciflow/all/69211 -> ciflow/all/69211 2022-03-17T19:04:59.1749175Z * [new tag] ciflow/all/69754 -> ciflow/all/69754 2022-03-17T19:04:59.1749847Z * [new tag] ciflow/all/69828 -> ciflow/all/69828 2022-03-17T19:04:59.1750544Z * [new tag] ciflow/all/70610 -> ciflow/all/70610 2022-03-17T19:04:59.1751547Z * [new tag] ciflow/all/70978 -> ciflow/all/70978 2022-03-17T19:04:59.1752245Z * [new tag] ciflow/all/70979 -> ciflow/all/70979 2022-03-17T19:04:59.1752924Z * [new tag] ciflow/all/70980 -> ciflow/all/70980 2022-03-17T19:04:59.1753559Z * [new tag] ciflow/all/70981 -> ciflow/all/70981 2022-03-17T19:04:59.1754184Z * [new tag] ciflow/all/70982 -> ciflow/all/70982 2022-03-17T19:04:59.1754851Z * [new tag] ciflow/all/70986 -> ciflow/all/70986 2022-03-17T19:04:59.1755568Z * [new tag] ciflow/all/70988 -> ciflow/all/70988 2022-03-17T19:04:59.1756271Z * [new tag] ciflow/all/70989 -> ciflow/all/70989 2022-03-17T19:04:59.1756956Z * [new tag] ciflow/all/71399 -> ciflow/all/71399 2022-03-17T19:04:59.1757644Z * [new tag] ciflow/all/71582 -> ciflow/all/71582 2022-03-17T19:04:59.1758329Z * [new tag] ciflow/all/71670 -> ciflow/all/71670 2022-03-17T19:04:59.1759063Z * [new tag] ciflow/all/72364 -> ciflow/all/72364 2022-03-17T19:04:59.1759762Z * [new tag] ciflow/all/72593 -> ciflow/all/72593 2022-03-17T19:04:59.1760454Z * [new tag] ciflow/all/72710 -> ciflow/all/72710 2022-03-17T19:04:59.1761152Z * [new tag] ciflow/all/73202 -> ciflow/all/73202 2022-03-17T19:04:59.1762087Z * [new tag] ciflow/all/73403 -> ciflow/all/73403 2022-03-17T19:04:59.1762669Z * [new tag] ciflow/all/73627 -> ciflow/all/73627 2022-03-17T19:04:59.1764043Z * [new tag] ciflow/all/73660 -> ciflow/all/73660 2022-03-17T19:04:59.1764766Z * [new tag] ciflow/all/74111 -> ciflow/all/74111 2022-03-17T19:04:59.1765463Z * [new tag] ciflow/all/74226 -> ciflow/all/74226 2022-03-17T19:04:59.1766201Z * [new tag] ciflow/binaries/71893 -> ciflow/binaries/71893 2022-03-17T19:04:59.1766957Z * [new tag] ciflow/binaries/73342 -> ciflow/binaries/73342 2022-03-17T19:04:59.1767736Z * [new tag] ciflow/binaries/73393 -> ciflow/binaries/73393 2022-03-17T19:04:59.1768499Z * [new tag] ciflow/binaries/74141 -> ciflow/binaries/74141 2022-03-17T19:04:59.1769248Z * [new tag] ciflow/cpu/67093 -> ciflow/cpu/67093 2022-03-17T19:04:59.1769961Z * [new tag] ciflow/cpu/67094 -> ciflow/cpu/67094 2022-03-17T19:04:59.1770692Z * [new tag] ciflow/cpu/67096 -> ciflow/cpu/67096 2022-03-17T19:04:59.1771371Z * [new tag] ciflow/cpu/71399 -> ciflow/cpu/71399 2022-03-17T19:04:59.1771995Z * [new tag] ciflow/cpu/73342 -> ciflow/cpu/73342 2022-03-17T19:04:59.1773022Z * [new tag] ciflow/cpu/74294 -> ciflow/cpu/74294 2022-03-17T19:04:59.1773828Z * [new tag] ciflow/cuda/69828 -> ciflow/cuda/69828 2022-03-17T19:04:59.1774497Z * [new tag] ciflow/cuda/70978 -> ciflow/cuda/70978 2022-03-17T19:04:59.1774831Z * [new tag] ciflow/cuda/70979 -> ciflow/cuda/70979 2022-03-17T19:04:59.1775163Z * [new tag] ciflow/cuda/70980 -> ciflow/cuda/70980 2022-03-17T19:04:59.1775453Z * [new tag] ciflow/cuda/70981 -> ciflow/cuda/70981 2022-03-17T19:04:59.1775787Z * [new tag] ciflow/cuda/70982 -> ciflow/cuda/70982 2022-03-17T19:04:59.1776076Z * [new tag] ciflow/cuda/70986 -> ciflow/cuda/70986 2022-03-17T19:04:59.1776665Z * [new tag] ciflow/cuda/70988 -> ciflow/cuda/70988 2022-03-17T19:04:59.1777362Z * [new tag] ciflow/cuda/70989 -> ciflow/cuda/70989 2022-03-17T19:04:59.1777953Z * [new tag] ciflow/cuda/71213 -> ciflow/cuda/71213 2022-03-17T19:04:59.1778598Z * [new tag] ciflow/cuda/71214 -> ciflow/cuda/71214 2022-03-17T19:04:59.1779470Z * [new tag] ciflow/cuda/71215 -> ciflow/cuda/71215 2022-03-17T19:04:59.1779982Z * [new tag] ciflow/cuda/71216 -> ciflow/cuda/71216 2022-03-17T19:04:59.1780368Z * [new tag] ciflow/cuda/71399 -> ciflow/cuda/71399 2022-03-17T19:04:59.1781094Z * [new tag] ciflow/cuda/74294 -> ciflow/cuda/74294 2022-03-17T19:04:59.1781950Z * [new tag] ciflow/cuda/74343 -> ciflow/cuda/74343 2022-03-17T19:04:59.1782855Z * [new tag] ciflow/ios/72666 -> ciflow/ios/72666 2022-03-17T19:04:59.1783780Z * [new tag] ciflow/libtorch/73011 -> ciflow/libtorch/73011 2022-03-17T19:04:59.1784517Z * [new tag] ciflow/macos/73849 -> ciflow/macos/73849 2022-03-17T19:04:59.1785143Z * [new tag] ciflow/macos/73857 -> ciflow/macos/73857 2022-03-17T19:04:59.1785983Z * [new tag] ciflow/none/73275 -> ciflow/none/73275 2022-03-17T19:04:59.1786787Z * [new tag] ciflow/periodic/74306 -> ciflow/periodic/74306 2022-03-17T19:04:59.1787557Z * [new tag] ciflow/slow-gradcheck/71126 -> ciflow/slow-gradcheck/71126 2022-03-17T19:04:59.1788194Z * [new tag] ciflow/slow-gradcheck/73748 -> ciflow/slow-gradcheck/73748 2022-03-17T19:04:59.1788866Z * [new tag] ciflow/slow/71126 -> ciflow/slow/71126 2022-03-17T19:04:59.1789467Z * [new tag] ciflow/slow/72686 -> ciflow/slow/72686 2022-03-17T19:04:59.1790103Z * [new tag] ciflow/slow/73627 -> ciflow/slow/73627 2022-03-17T19:04:59.1790702Z * [new tag] ciflow/slow/73748 -> ciflow/slow/73748 2022-03-17T19:04:59.1791645Z * [new tag] ciflow/trunk/73788 -> ciflow/trunk/73788 2022-03-17T19:04:59.1792249Z * [new tag] ciflow/trunk/74306 -> ciflow/trunk/74306 2022-03-17T19:04:59.1793036Z * [new tag] nightly-binary -> nightly-binary 2022-03-17T19:04:59.1793764Z * [new tag] v0.1.1 -> v0.1.1 2022-03-17T19:04:59.1794593Z * [new tag] v0.1.10 -> v0.1.10 2022-03-17T19:04:59.1795285Z * [new tag] v0.1.11 -> v0.1.11 2022-03-17T19:04:59.1796018Z * [new tag] v0.1.12 -> v0.1.12 2022-03-17T19:04:59.1796744Z * [new tag] v0.1.2 -> v0.1.2 2022-03-17T19:04:59.1797556Z * [new tag] v0.1.3 -> v0.1.3 2022-03-17T19:04:59.1798343Z * [new tag] v0.1.4 -> v0.1.4 2022-03-17T19:04:59.1799255Z * [new tag] v0.1.5 -> v0.1.5 2022-03-17T19:04:59.1799868Z * [new tag] v0.1.6 -> v0.1.6 2022-03-17T19:04:59.1800615Z * [new tag] v0.1.7 -> v0.1.7 2022-03-17T19:04:59.1801316Z * [new tag] v0.1.8 -> v0.1.8 2022-03-17T19:04:59.1802316Z * [new tag] v0.1.9 -> v0.1.9 2022-03-17T19:04:59.1803185Z * [new tag] v0.2.0 -> v0.2.0 2022-03-17T19:04:59.1803928Z * [new tag] v0.3.0 -> v0.3.0 2022-03-17T19:04:59.1804791Z * [new tag] v0.3.1 -> v0.3.1 2022-03-17T19:04:59.1805611Z * [new tag] v0.4.0 -> v0.4.0 2022-03-17T19:04:59.1806298Z * [new tag] v0.4.1 -> v0.4.1 2022-03-17T19:04:59.1807053Z * [new tag] v1.0.0 -> v1.0.0 2022-03-17T19:04:59.1807898Z * [new tag] v1.0.0a0 -> v1.0.0a0 2022-03-17T19:04:59.1808770Z * [new tag] v1.0.1 -> v1.0.1 2022-03-17T19:04:59.1809542Z * [new tag] v1.0rc0 -> v1.0rc0 2022-03-17T19:04:59.1810142Z * [new tag] v1.0rc1 -> v1.0rc1 2022-03-17T19:04:59.1810895Z * [new tag] v1.1.0 -> v1.1.0 2022-03-17T19:04:59.1811708Z * [new tag] v1.1.0a0 -> v1.1.0a0 2022-03-17T19:04:59.1812609Z * [new tag] v1.10.0 -> v1.10.0 2022-03-17T19:04:59.1813510Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2022-03-17T19:04:59.1814297Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2022-03-17T19:04:59.1814891Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2022-03-17T19:04:59.1815665Z * [new tag] v1.10.1 -> v1.10.1 2022-03-17T19:04:59.1816266Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2022-03-17T19:04:59.1816860Z * [new tag] v1.10.2 -> v1.10.2 2022-03-17T19:04:59.1817425Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2022-03-17T19:04:59.1818180Z * [new tag] v1.11.0 -> v1.11.0 2022-03-17T19:04:59.1819044Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2022-03-17T19:04:59.1819945Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2022-03-17T19:04:59.1820772Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2022-03-17T19:04:59.1821592Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2022-03-17T19:04:59.1822440Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2022-03-17T19:04:59.1823071Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2022-03-17T19:04:59.1823649Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2022-03-17T19:04:59.1824335Z * [new tag] v1.2.0 -> v1.2.0 2022-03-17T19:04:59.1825204Z * [new tag] v1.2.0a0 -> v1.2.0a0 2022-03-17T19:04:59.1825936Z * [new tag] v1.3.0 -> v1.3.0 2022-03-17T19:04:59.1826706Z * [new tag] v1.3.0a0 -> v1.3.0a0 2022-03-17T19:04:59.1827317Z * [new tag] v1.3.1 -> v1.3.1 2022-03-17T19:04:59.1828022Z * [new tag] v1.4.0 -> v1.4.0 2022-03-17T19:04:59.1828811Z * [new tag] v1.4.0a0 -> v1.4.0a0 2022-03-17T19:04:59.1829410Z * [new tag] v1.4.1 -> v1.4.1 2022-03-17T19:04:59.1830222Z * [new tag] v1.5.0 -> v1.5.0 2022-03-17T19:04:59.1831067Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2022-03-17T19:04:59.1832105Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2022-03-17T19:04:59.1832751Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2022-03-17T19:04:59.1833481Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2022-03-17T19:04:59.1834089Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2022-03-17T19:04:59.1834899Z * [new tag] v1.5.1 -> v1.5.1 2022-03-17T19:04:59.1835526Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2022-03-17T19:04:59.1836137Z * [new tag] v1.6.0 -> v1.6.0 2022-03-17T19:04:59.1837002Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2022-03-17T19:04:59.1837878Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2022-03-17T19:04:59.1838693Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2022-03-17T19:04:59.1839495Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2022-03-17T19:04:59.1840465Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2022-03-17T19:04:59.1841266Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2022-03-17T19:04:59.1842120Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2022-03-17T19:04:59.1842669Z * [new tag] v1.7.0 -> v1.7.0 2022-03-17T19:04:59.1843548Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2022-03-17T19:04:59.1844378Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2022-03-17T19:04:59.1845169Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2022-03-17T19:04:59.1845793Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2022-03-17T19:04:59.1846570Z * [new tag] v1.7.1 -> v1.7.1 2022-03-17T19:04:59.1847491Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2022-03-17T19:04:59.1848203Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2022-03-17T19:04:59.1848749Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2022-03-17T19:04:59.1849664Z * [new tag] v1.8.0 -> v1.8.0 2022-03-17T19:04:59.1850349Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2022-03-17T19:04:59.1851101Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2022-03-17T19:04:59.1851893Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2022-03-17T19:04:59.1852676Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2022-03-17T19:04:59.1853273Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2022-03-17T19:04:59.1853876Z * [new tag] v1.8.1 -> v1.8.1 2022-03-17T19:04:59.1854908Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2022-03-17T19:04:59.1855837Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2022-03-17T19:04:59.1856383Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2022-03-17T19:04:59.1857639Z * [new tag] v1.8.2 -> v1.8.2 2022-03-17T19:04:59.1858250Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2022-03-17T19:04:59.1859023Z * [new tag] v1.9.0 -> v1.9.0 2022-03-17T19:04:59.1859976Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2022-03-17T19:04:59.1860510Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2022-03-17T19:04:59.1861506Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2022-03-17T19:04:59.1862124Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2022-03-17T19:04:59.1862829Z * [new tag] v1.9.1 -> v1.9.1 2022-03-17T19:04:59.1863967Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2022-03-17T19:04:59.1864769Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2022-03-17T19:04:59.2815736Z [command]/usr/bin/git rev-parse --verify --quiet 4b4f652f799c732a7280052e44a803a4cf6084de^{object} 2022-03-17T19:04:59.2851556Z 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:04:59.2856083Z ##[endgroup] 2022-03-17T19:04:59.2857202Z ##[group]Determining the checkout info 2022-03-17T19:04:59.2859788Z ##[endgroup] 2022-03-17T19:04:59.2860973Z ##[group]Checking out the ref 2022-03-17T19:04:59.2871472Z [command]/usr/bin/git checkout --progress --force 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:05:00.0355524Z Note: switching to '4b4f652f799c732a7280052e44a803a4cf6084de'. 2022-03-17T19:05:00.0356118Z 2022-03-17T19:05:00.0356898Z You are in 'detached HEAD' state. You can look around, make experimental 2022-03-17T19:05:00.0357903Z changes and commit them, and you can discard any commits you make in this 2022-03-17T19:05:00.0358871Z state without impacting any branches by switching back to a branch. 2022-03-17T19:05:00.0359411Z 2022-03-17T19:05:00.0360123Z If you want to create a new branch to retain commits you create, you may 2022-03-17T19:05:00.0361098Z do so (now or later) by using -c with the switch command. Example: 2022-03-17T19:05:00.0361514Z 2022-03-17T19:05:00.0362048Z git switch -c 2022-03-17T19:05:00.0362429Z 2022-03-17T19:05:00.0362663Z Or undo this operation with: 2022-03-17T19:05:00.0362975Z 2022-03-17T19:05:00.0363162Z git switch - 2022-03-17T19:05:00.0363438Z 2022-03-17T19:05:00.0363831Z Turn off this advice by setting config variable advice.detachedHead to false 2022-03-17T19:05:00.0364305Z 2022-03-17T19:05:00.0364639Z HEAD is now at 4b4f652f79 [3/5] Put JIT source inside flatbuffer (#74245) 2022-03-17T19:05:00.0440311Z ##[endgroup] 2022-03-17T19:05:00.0441490Z ##[group]Setting up auth for fetching submodules 2022-03-17T19:05:00.0457410Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/8e1ab627-9261-46a2-91f6-1f67f296ef61' before making global git config changes 2022-03-17T19:05:00.0459830Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-03-17T19:05:00.0513428Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-03-17T19:05:00.0557268Z [command]/usr/bin/git config --global url.https://github.com/.insteadOf git@github.com: 2022-03-17T19:05:00.0596449Z ##[endgroup] 2022-03-17T19:05:00.0597533Z ##[group]Fetching submodules 2022-03-17T19:05:00.0598739Z [command]/usr/bin/git submodule sync --recursive 2022-03-17T19:05:00.0989739Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-03-17T19:05:00.1374246Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-03-17T19:05:00.1376913Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-03-17T19:05:00.1383476Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-03-17T19:05:00.1389882Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-03-17T19:05:00.1397574Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-03-17T19:05:00.1405130Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-03-17T19:05:00.1412562Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-03-17T19:05:00.1417053Z Submodule 'third_party/breakpad' (https://github.com/driazati/breakpad.git) registered for path 'third_party/breakpad' 2022-03-17T19:05:00.1420079Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-03-17T19:05:00.1423400Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-03-17T19:05:00.1426479Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-03-17T19:05:00.1429729Z Submodule 'third_party/eigen' (https://github.com/eigenteam/eigen-git-mirror.git) registered for path 'third_party/eigen' 2022-03-17T19:05:00.1432908Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-03-17T19:05:00.1436243Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-03-17T19:05:00.1439510Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-03-17T19:05:00.1443325Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-03-17T19:05:00.1447727Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:05:00.1452042Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-03-17T19:05:00.1456119Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-03-17T19:05:00.1459787Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-03-17T19:05:00.1463638Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-03-17T19:05:00.1467475Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-03-17T19:05:00.1471462Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-03-17T19:05:00.1475644Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-03-17T19:05:00.1479665Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-03-17T19:05:00.1484287Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-03-17T19:05:00.1489379Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-03-17T19:05:00.1494644Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-03-17T19:05:00.1499093Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-03-17T19:05:00.1503776Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-03-17T19:05:00.1508157Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-03-17T19:05:00.1512747Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-03-17T19:05:00.1517561Z Submodule 'third_party/python-peachpy' (https://github.com/Maratyszcza/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-03-17T19:05:00.1522396Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-03-17T19:05:00.1528008Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-03-17T19:05:00.1533748Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-03-17T19:05:00.1539062Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-03-17T19:05:00.1544141Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-03-17T19:05:00.1658036Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-03-17T19:05:01.2565315Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-03-17T19:05:02.8910616Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-03-17T19:05:03.9440308Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-03-17T19:05:05.5083730Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-03-17T19:05:06.9685131Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-03-17T19:05:13.6925342Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-03-17T19:05:15.3113288Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/breakpad'... 2022-03-17T19:05:28.2755329Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-03-17T19:05:29.7328408Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-03-17T19:05:32.2017507Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-03-17T19:05:41.0965407Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-03-17T19:05:47.3227067Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-03-17T19:05:51.3358470Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-03-17T19:06:04.2906982Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-03-17T19:06:07.0747675Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-03-17T19:06:08.1514547Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-03-17T19:06:10.2737542Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-03-17T19:06:11.8192922Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-03-17T19:06:14.5977938Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-03-17T19:06:16.0734440Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-03-17T19:06:17.0213987Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-03-17T19:06:45.8686821Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-03-17T19:06:47.3385396Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-03-17T19:06:48.4970090Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-03-17T19:06:51.2727102Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-03-17T19:06:52.6662102Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-03-17T19:06:53.6095434Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-03-17T19:06:58.9876747Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-03-17T19:06:59.9640332Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-03-17T19:07:01.3549646Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-03-17T19:07:03.8099516Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-03-17T19:07:04.7275649Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-03-17T19:07:05.9432309Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-03-17T19:07:07.1965775Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-03-17T19:07:08.7929708Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-03-17T19:07:12.1158056Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-03-17T19:07:13.4681242Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-03-17T19:07:20.4311247Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-03-17T19:07:20.4713297Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-03-17T19:07:20.5058209Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-03-17T19:07:20.5612784Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-03-17T19:07:20.6148250Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-03-17T19:07:21.2012103Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-03-17T19:07:21.2589852Z Submodule path 'third_party/benchmark': checked out 'e991355c02b93fe17713efe04cbc2e278e00fdbd' 2022-03-17T19:07:21.4500691Z Submodule path 'third_party/breakpad': checked out '7d188f679d4ae0a5bd06408a3047d69ef8eef848' 2022-03-17T19:07:21.4578876Z Submodule 'src/third_party/lss' (https://chromium.googlesource.com/linux-syscall-support) registered for path 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:07:21.4659126Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/breakpad/src/third_party/lss'... 2022-03-17T19:07:22.3276576Z Submodule path 'third_party/breakpad/src/third_party/lss': checked out 'e1e7b0ad8ee99a875b272c8e33e308472e897660' 2022-03-17T19:07:22.4611247Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d' 2022-03-17T19:07:22.5346407Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-03-17T19:07:22.8341567Z Submodule path 'third_party/cudnn_frontend': checked out '7b83dba83fa31381aeca508d89aab94f4639ac6d' 2022-03-17T19:07:23.1733541Z Submodule path 'third_party/eigen': checked out 'd41dc4dd74acce21fb210e7625d5d135751fa9e5' 2022-03-17T19:07:23.2473474Z Submodule path 'third_party/fbgemm': checked out 'bedfcaeea242e7d900275d5650c744307d34ca03' 2022-03-17T19:07:23.2541910Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:07:23.2546343Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:07:23.2551775Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:07:23.2631048Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-03-17T19:07:25.9084260Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-03-17T19:07:27.3452677Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-03-17T19:07:29.4385204Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out '8b35b4cffb62ecb58a903bf91cb7537d7a672211' 2022-03-17T19:07:29.5709914Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-03-17T19:07:29.6733599Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-03-17T19:07:29.7898829Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-03-17T19:07:29.8589217Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2022-03-17T19:07:29.8953372Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-03-17T19:07:29.9647097Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-03-17T19:07:30.0194650Z Submodule path 'third_party/gloo': checked out 'c22a5cfba94edf8ea4f53a174d38aa0c629d070f' 2022-03-17T19:07:30.0968754Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-03-17T19:07:30.1385588Z Submodule path 'third_party/ideep': checked out '4a56ab2c3f61c44e0f8ea241beeb732b7d70dc5b' 2022-03-17T19:07:30.1460118Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-03-17T19:07:30.1536398Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-03-17T19:08:41.7403885Z Submodule path 'third_party/ideep/mkl-dnn': checked out '16548c949f392376996ff180c695908ece5f83f0' 2022-03-17T19:08:41.7473349Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:08:41.7544014Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-03-17T19:10:35.2551999Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'a9302535553c73243c632ad3c4c80beec3d19a1e' 2022-03-17T19:10:35.2955003Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-03-17T19:10:35.4176419Z Submodule path 'third_party/kineto': checked out '422591614ae4ad2cbc85ea24de48727e8ec12481' 2022-03-17T19:10:35.4234680Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:10:35.4239225Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:10:35.4315176Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-03-17T19:10:37.3523054Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-03-17T19:10:40.1288658Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-03-17T19:10:40.2200187Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-03-17T19:10:40.2721393Z Submodule path 'third_party/nccl/nccl': checked out '7e515921295adaab72adf56ea71a0fafb0ecb5f3' 2022-03-17T19:10:40.3168606Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-03-17T19:10:40.5707174Z Submodule path 'third_party/onnx': checked out '96046b8ccfb8e6fa82f6b2b34b3d56add2e8849c' 2022-03-17T19:10:40.5810587Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-03-17T19:10:40.5815593Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-03-17T19:10:40.5919164Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-03-17T19:10:44.1751988Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-03-17T19:10:51.0955467Z Submodule path 'third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-03-17T19:10:51.1631371Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '59a2ac2745d8a57ac94c6accced73620d59fb844' 2022-03-17T19:10:51.2123680Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-03-17T19:10:51.2189157Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:10:51.2263322Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-03-17T19:10:54.6164541Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-03-17T19:10:54.6255060Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:10:54.6259333Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:10:54.6356070Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-03-17T19:10:56.1264338Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-03-17T19:11:04.1187174Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-03-17T19:11:04.2315447Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-03-17T19:11:04.2399010Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:11:04.2473366Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-03-17T19:11:05.3253729Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-03-17T19:11:05.3653828Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-03-17T19:11:05.6506227Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-03-17T19:11:05.6736032Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:11:05.6749073Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-03-17T19:11:05.6835636Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-03-17T19:11:07.4617066Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-03-17T19:11:09.6127657Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-03-17T19:11:09.7224201Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-03-17T19:11:09.7603067Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-03-17T19:11:09.8000186Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-03-17T19:11:09.8644893Z Submodule path 'third_party/pybind11': checked out '8de7772cc72daca8e947b79b83fea46214931604' 2022-03-17T19:11:09.9026283Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-03-17T19:11:09.9613998Z Submodule path 'third_party/python-peachpy': checked out '07d8fde8ac45d7705129475c0f94ed8925b93473' 2022-03-17T19:11:10.0001273Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-03-17T19:11:10.0748845Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-03-17T19:11:10.2186948Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-03-17T19:11:10.2774654Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-03-17T19:11:10.2848916Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:11:10.2854376Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:11:10.2861140Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:11:10.2869460Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:11:10.2953248Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-03-17T19:11:14.7432452Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-03-17T19:11:15.7695612Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-03-17T19:11:18.0356485Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-03-17T19:11:20.6224735Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-03-17T19:11:20.6663955Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-03-17T19:11:20.7778605Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-03-17T19:11:20.8398684Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-03-17T19:11:20.8474007Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:11:20.8550187Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-03-17T19:11:21.8581184Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-03-17T19:11:22.0403557Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-03-17T19:11:22.0538034Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-03-17T19:11:22.0943270Z Entering 'android/libs/fbjni' 2022-03-17T19:11:22.1002171Z Entering 'third_party/FP16' 2022-03-17T19:11:22.1072484Z Entering 'third_party/FXdiv' 2022-03-17T19:11:22.1140033Z Entering 'third_party/NNPACK' 2022-03-17T19:11:22.1205261Z Entering 'third_party/QNNPACK' 2022-03-17T19:11:22.1277633Z Entering 'third_party/XNNPACK' 2022-03-17T19:11:22.1367562Z Entering 'third_party/benchmark' 2022-03-17T19:11:22.1427831Z Entering 'third_party/breakpad' 2022-03-17T19:11:22.1500036Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:11:22.1555956Z Entering 'third_party/cpuinfo' 2022-03-17T19:11:22.1628506Z Entering 'third_party/cub' 2022-03-17T19:11:22.1687568Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:11:22.1765957Z Entering 'third_party/eigen' 2022-03-17T19:11:22.1841517Z Entering 'third_party/fbgemm' 2022-03-17T19:11:22.1906834Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:11:22.1977170Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:11:22.2050432Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:11:22.2117614Z Entering 'third_party/flatbuffers' 2022-03-17T19:11:22.2189767Z Entering 'third_party/fmt' 2022-03-17T19:11:22.2261271Z Entering 'third_party/foxi' 2022-03-17T19:11:22.2331911Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:11:22.2401599Z Entering 'third_party/gloo' 2022-03-17T19:11:22.2471493Z Entering 'third_party/googletest' 2022-03-17T19:11:22.2529826Z Entering 'third_party/ideep' 2022-03-17T19:11:22.2591244Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:11:22.2662693Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:11:22.2747737Z Entering 'third_party/ios-cmake' 2022-03-17T19:11:22.2809749Z Entering 'third_party/kineto' 2022-03-17T19:11:22.2878428Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:11:22.2944633Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:11:22.3010096Z Entering 'third_party/nccl/nccl' 2022-03-17T19:11:22.3080951Z Entering 'third_party/neon2sse' 2022-03-17T19:11:22.3151063Z Entering 'third_party/onnx' 2022-03-17T19:11:22.3244421Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:11:22.3311644Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:11:22.3389026Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:11:22.3458831Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:11:22.3541280Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:11:22.3606345Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:11:22.3674604Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:11:22.3761640Z Entering 'third_party/pocketfft' 2022-03-17T19:11:22.3828254Z Entering 'third_party/protobuf' 2022-03-17T19:11:22.3895485Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:11:22.3969930Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:11:22.4043396Z Entering 'third_party/psimd' 2022-03-17T19:11:22.4113942Z Entering 'third_party/pthreadpool' 2022-03-17T19:11:22.4183806Z Entering 'third_party/pybind11' 2022-03-17T19:11:22.4254384Z Entering 'third_party/python-enum' 2022-03-17T19:11:22.4326328Z Entering 'third_party/python-peachpy' 2022-03-17T19:11:22.4393559Z Entering 'third_party/python-six' 2022-03-17T19:11:22.4463609Z Entering 'third_party/sleef' 2022-03-17T19:11:22.4533982Z Entering 'third_party/tbb' 2022-03-17T19:11:22.4600735Z Entering 'third_party/tensorpipe' 2022-03-17T19:11:22.4667120Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:11:22.4738670Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:11:22.4802915Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:11:22.4861759Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:11:22.4934564Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:11:22.5004740Z Entering 'third_party/zstd' 2022-03-17T19:11:22.5091062Z ##[endgroup] 2022-03-17T19:11:22.5092230Z ##[group]Persisting credentials for submodules 2022-03-17T19:11:22.5108690Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-03-17T19:11:22.5520421Z Entering 'android/libs/fbjni' 2022-03-17T19:11:22.5579965Z Entering 'third_party/FP16' 2022-03-17T19:11:22.5646945Z Entering 'third_party/FXdiv' 2022-03-17T19:11:22.5714474Z Entering 'third_party/NNPACK' 2022-03-17T19:11:22.5777187Z Entering 'third_party/QNNPACK' 2022-03-17T19:11:22.5846546Z Entering 'third_party/XNNPACK' 2022-03-17T19:11:22.5933091Z Entering 'third_party/benchmark' 2022-03-17T19:11:22.6000959Z Entering 'third_party/breakpad' 2022-03-17T19:11:22.6068739Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:11:22.6138219Z Entering 'third_party/cpuinfo' 2022-03-17T19:11:22.6196817Z Entering 'third_party/cub' 2022-03-17T19:11:22.6258650Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:11:22.6335175Z Entering 'third_party/eigen' 2022-03-17T19:11:22.6403687Z Entering 'third_party/fbgemm' 2022-03-17T19:11:22.6470388Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:11:22.6531714Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:11:22.6592039Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:11:22.6659492Z Entering 'third_party/flatbuffers' 2022-03-17T19:11:22.6721336Z Entering 'third_party/fmt' 2022-03-17T19:11:22.6789590Z Entering 'third_party/foxi' 2022-03-17T19:11:22.6858917Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:11:22.6928066Z Entering 'third_party/gloo' 2022-03-17T19:11:22.6991742Z Entering 'third_party/googletest' 2022-03-17T19:11:22.7062157Z Entering 'third_party/ideep' 2022-03-17T19:11:22.7126979Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:11:22.7183776Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:11:22.7256126Z Entering 'third_party/ios-cmake' 2022-03-17T19:11:22.7314024Z Entering 'third_party/kineto' 2022-03-17T19:11:22.7376028Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:11:22.7435178Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:11:22.7509392Z Entering 'third_party/nccl/nccl' 2022-03-17T19:11:22.7574512Z Entering 'third_party/neon2sse' 2022-03-17T19:11:22.7636170Z Entering 'third_party/onnx' 2022-03-17T19:11:22.7728313Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:11:22.7797957Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:11:22.7865487Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:11:22.7934815Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:11:22.8014568Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:11:22.8073375Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:11:22.8133391Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:11:22.8216917Z Entering 'third_party/pocketfft' 2022-03-17T19:11:22.8283713Z Entering 'third_party/protobuf' 2022-03-17T19:11:22.8356193Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:11:22.8425755Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:11:22.8489597Z Entering 'third_party/psimd' 2022-03-17T19:11:22.8552954Z Entering 'third_party/pthreadpool' 2022-03-17T19:11:22.8618320Z Entering 'third_party/pybind11' 2022-03-17T19:11:22.8686407Z Entering 'third_party/python-enum' 2022-03-17T19:11:22.8747887Z Entering 'third_party/python-peachpy' 2022-03-17T19:11:22.8807817Z Entering 'third_party/python-six' 2022-03-17T19:11:22.8875426Z Entering 'third_party/sleef' 2022-03-17T19:11:22.8944016Z Entering 'third_party/tbb' 2022-03-17T19:11:22.9017328Z Entering 'third_party/tensorpipe' 2022-03-17T19:11:22.9088389Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:11:22.9155249Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:11:22.9220672Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:11:22.9288774Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:11:22.9354938Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:11:22.9431905Z Entering 'third_party/zstd' 2022-03-17T19:11:22.9520467Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-03-17T19:11:22.9924542Z Entering 'android/libs/fbjni' 2022-03-17T19:11:22.9972730Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-03-17T19:11:23.0006016Z Entering 'third_party/FP16' 2022-03-17T19:11:23.0064666Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-03-17T19:11:23.0096794Z Entering 'third_party/FXdiv' 2022-03-17T19:11:23.0153493Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-03-17T19:11:23.0187394Z Entering 'third_party/NNPACK' 2022-03-17T19:11:23.0244030Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-03-17T19:11:23.0276919Z Entering 'third_party/QNNPACK' 2022-03-17T19:11:23.0340450Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-03-17T19:11:23.0374224Z Entering 'third_party/XNNPACK' 2022-03-17T19:11:23.0434430Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-03-17T19:11:23.0486367Z Entering 'third_party/benchmark' 2022-03-17T19:11:23.0542199Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:11:23.0576344Z Entering 'third_party/breakpad' 2022-03-17T19:11:23.0635443Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/breakpad/config remote.origin.url 2022-03-17T19:11:23.0671284Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:11:23.0734676Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/breakpad/modules/src/third_party/lss/config remote.origin.url 2022-03-17T19:11:23.0770725Z Entering 'third_party/cpuinfo' 2022-03-17T19:11:23.0831136Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-03-17T19:11:23.0866304Z Entering 'third_party/cub' 2022-03-17T19:11:23.0927245Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-03-17T19:11:23.0951326Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:11:23.1006951Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-03-17T19:11:23.1049952Z Entering 'third_party/eigen' 2022-03-17T19:11:23.1105264Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-03-17T19:11:23.1142608Z Entering 'third_party/fbgemm' 2022-03-17T19:11:23.1200180Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-03-17T19:11:23.1236191Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:11:23.1287612Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-03-17T19:11:23.1320164Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:11:23.1381986Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-03-17T19:11:23.1412258Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:11:23.1470132Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:11:23.1507652Z Entering 'third_party/flatbuffers' 2022-03-17T19:11:23.1561766Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-03-17T19:11:23.1599388Z Entering 'third_party/fmt' 2022-03-17T19:11:23.1660656Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-03-17T19:11:23.1687516Z Entering 'third_party/foxi' 2022-03-17T19:11:23.1743445Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-03-17T19:11:23.1780305Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:11:23.1838073Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-03-17T19:11:23.1870055Z Entering 'third_party/gloo' 2022-03-17T19:11:23.1926679Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-03-17T19:11:23.1960040Z Entering 'third_party/googletest' 2022-03-17T19:11:23.2018809Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:11:23.2050361Z Entering 'third_party/ideep' 2022-03-17T19:11:23.2110051Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-03-17T19:11:23.2134000Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:11:23.2194732Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-03-17T19:11:23.2231563Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:11:23.2296592Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-03-17T19:11:23.2338786Z Entering 'third_party/ios-cmake' 2022-03-17T19:11:23.2399991Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-03-17T19:11:23.2432318Z Entering 'third_party/kineto' 2022-03-17T19:11:23.2486176Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-03-17T19:11:23.2518355Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:11:23.2577840Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-03-17T19:11:23.2607453Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:11:23.2666151Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-03-17T19:11:23.2700985Z Entering 'third_party/nccl/nccl' 2022-03-17T19:11:23.2761633Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-03-17T19:11:23.2792817Z Entering 'third_party/neon2sse' 2022-03-17T19:11:23.2847765Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-03-17T19:11:23.2881081Z Entering 'third_party/onnx' 2022-03-17T19:11:23.2942192Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-03-17T19:11:23.2999202Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:11:23.3044245Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:11:23.3077380Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:11:23.3135897Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:11:23.3172275Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:11:23.3226559Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-03-17T19:11:23.3259959Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:11:23.3322229Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-03-17T19:11:23.3367099Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:11:23.3430640Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:11:23.3460919Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:11:23.3526128Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:11:23.3557769Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:11:23.3620225Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-03-17T19:11:23.3664163Z Entering 'third_party/pocketfft' 2022-03-17T19:11:23.3719875Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-03-17T19:11:23.3752369Z Entering 'third_party/protobuf' 2022-03-17T19:11:23.3812977Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-03-17T19:11:23.3853506Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:11:23.3902524Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:11:23.3933557Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:11:23.3993368Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:11:23.4032434Z Entering 'third_party/psimd' 2022-03-17T19:11:23.4090506Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-03-17T19:11:23.4116512Z Entering 'third_party/pthreadpool' 2022-03-17T19:11:23.4178685Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-03-17T19:11:23.4211592Z Entering 'third_party/pybind11' 2022-03-17T19:11:23.4265577Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:11:23.4302094Z Entering 'third_party/python-enum' 2022-03-17T19:11:23.4355399Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-03-17T19:11:23.4392463Z Entering 'third_party/python-peachpy' 2022-03-17T19:11:23.4454520Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-03-17T19:11:23.4482612Z Entering 'third_party/python-six' 2022-03-17T19:11:23.4536009Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-03-17T19:11:23.4568556Z Entering 'third_party/sleef' 2022-03-17T19:11:23.4629207Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-03-17T19:11:23.4662910Z Entering 'third_party/tbb' 2022-03-17T19:11:23.4719166Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-03-17T19:11:23.4756878Z Entering 'third_party/tensorpipe' 2022-03-17T19:11:23.4821243Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-03-17T19:11:23.4852515Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:11:23.4914198Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:11:23.4943323Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:11:23.5000774Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-03-17T19:11:23.5033776Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:11:23.5091969Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-03-17T19:11:23.5125379Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:11:23.5179902Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:11:23.5214838Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:11:23.5275446Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-03-17T19:11:23.5316159Z Entering 'third_party/zstd' 2022-03-17T19:11:23.5378018Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-03-17T19:11:23.5754201Z [command]/usr/bin/git submodule foreach --recursive git config --local 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-03-17T19:11:23.6138195Z Entering 'android/libs/fbjni' 2022-03-17T19:11:23.6202574Z Entering 'third_party/FP16' 2022-03-17T19:11:23.6275854Z Entering 'third_party/FXdiv' 2022-03-17T19:11:23.6347190Z Entering 'third_party/NNPACK' 2022-03-17T19:11:23.6414708Z Entering 'third_party/QNNPACK' 2022-03-17T19:11:23.6479540Z Entering 'third_party/XNNPACK' 2022-03-17T19:11:23.6571046Z Entering 'third_party/benchmark' 2022-03-17T19:11:23.6645121Z Entering 'third_party/breakpad' 2022-03-17T19:11:23.6708480Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:11:23.6785904Z Entering 'third_party/cpuinfo' 2022-03-17T19:11:23.6852510Z Entering 'third_party/cub' 2022-03-17T19:11:23.6906608Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:11:23.6985682Z Entering 'third_party/eigen' 2022-03-17T19:11:23.7057334Z Entering 'third_party/fbgemm' 2022-03-17T19:11:23.7119817Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:11:23.7192794Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:11:23.7259906Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:11:23.7330652Z Entering 'third_party/flatbuffers' 2022-03-17T19:11:23.7400724Z Entering 'third_party/fmt' 2022-03-17T19:11:23.7465414Z Entering 'third_party/foxi' 2022-03-17T19:11:23.7529734Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:11:23.7592872Z Entering 'third_party/gloo' 2022-03-17T19:11:23.7664993Z Entering 'third_party/googletest' 2022-03-17T19:11:23.7729220Z Entering 'third_party/ideep' 2022-03-17T19:11:23.7794263Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:11:23.7866341Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:11:23.7952946Z Entering 'third_party/ios-cmake' 2022-03-17T19:11:23.8023720Z Entering 'third_party/kineto' 2022-03-17T19:11:23.8097263Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:11:23.8162680Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:11:23.8237517Z Entering 'third_party/nccl/nccl' 2022-03-17T19:11:23.8309149Z Entering 'third_party/neon2sse' 2022-03-17T19:11:23.8380500Z Entering 'third_party/onnx' 2022-03-17T19:11:23.8464324Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:11:23.8536063Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:11:23.8606146Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:11:23.8669518Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:11:23.8754621Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:11:23.8823871Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:11:23.8888177Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:11:23.8958667Z Entering 'third_party/pocketfft' 2022-03-17T19:11:23.9029396Z Entering 'third_party/protobuf' 2022-03-17T19:11:23.9108637Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:11:23.9177273Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:11:23.9251485Z Entering 'third_party/psimd' 2022-03-17T19:11:23.9310052Z Entering 'third_party/pthreadpool' 2022-03-17T19:11:23.9380589Z Entering 'third_party/pybind11' 2022-03-17T19:11:23.9448372Z Entering 'third_party/python-enum' 2022-03-17T19:11:23.9520626Z Entering 'third_party/python-peachpy' 2022-03-17T19:11:23.9584848Z Entering 'third_party/python-six' 2022-03-17T19:11:23.9645756Z Entering 'third_party/sleef' 2022-03-17T19:11:23.9716335Z Entering 'third_party/tbb' 2022-03-17T19:11:23.9790961Z Entering 'third_party/tensorpipe' 2022-03-17T19:11:23.9862496Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:11:23.9937816Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:11:23.9995606Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:11:24.0060523Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:11:24.0125473Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:11:24.0203285Z Entering 'third_party/zstd' 2022-03-17T19:11:24.0286768Z ##[endgroup] 2022-03-17T19:11:24.0441706Z [command]/usr/bin/git log -1 --format='%H' 2022-03-17T19:11:24.0495209Z '4b4f652f799c732a7280052e44a803a4cf6084de' 2022-03-17T19:11:24.0634633Z ##[group]Run # Remove any artifacts from the previous checkouts 2022-03-17T19:11:24.0635512Z # Remove any artifacts from the previous checkouts 2022-03-17T19:11:24.0636048Z git clean -fxd 2022-03-17T19:11:24.0693141Z shell: /bin/bash -e {0} 2022-03-17T19:11:24.0693551Z env: 2022-03-17T19:11:24.0694076Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:11:24.0694932Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:11:24.0695746Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:11:24.0696460Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:11:24.0697090Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:11:24.0697520Z IN_CI: 1 2022-03-17T19:11:24.0697900Z IS_GHA: 1 2022-03-17T19:11:24.0698295Z IN_WHEEL_TEST: 1 2022-03-17T19:11:24.0698817Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:11:24.0699540Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:11:24.0700125Z PR_LABELS: [] 2022-03-17T19:11:24.0701110Z GITHUB_TOKEN: *** 2022-03-17T19:11:24.0701576Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:11:24.0702034Z PR_NUMBER: 2022-03-17T19:11:24.0702491Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:11:24.0702972Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:11:24.0703456Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:11:24.0704273Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:11:24.0705221Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:11:24.0705801Z TEST_CONFIG: distributed 2022-03-17T19:11:24.0706223Z SHARD_NUMBER: 1 2022-03-17T19:11:24.0706649Z NUM_TEST_SHARDS: 1 2022-03-17T19:11:24.0707054Z PR_BODY: 2022-03-17T19:11:24.0707536Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:11:24.0708004Z ##[endgroup] 2022-03-17T19:11:24.1071197Z ##[group]Run retry () { 2022-03-17T19:11:24.1071768Z retry () { 2022-03-17T19:11:24.1072306Z  "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") 2022-03-17T19:11:24.1072817Z } 2022-03-17T19:11:24.1073359Z retry docker pull "${DOCKER_IMAGE}" 2022-03-17T19:11:24.1140743Z shell: /bin/bash -e {0} 2022-03-17T19:11:24.1140956Z env: 2022-03-17T19:11:24.1141232Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:11:24.1141675Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:11:24.1142109Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:11:24.1142489Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:11:24.1142816Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:11:24.1143032Z IN_CI: 1 2022-03-17T19:11:24.1143229Z IS_GHA: 1 2022-03-17T19:11:24.1143426Z IN_WHEEL_TEST: 1 2022-03-17T19:11:24.1143729Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:11:24.1144080Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:11:24.1144373Z PR_LABELS: [] 2022-03-17T19:11:24.1144861Z GITHUB_TOKEN: *** 2022-03-17T19:11:24.1145093Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:11:24.1145327Z PR_NUMBER: 2022-03-17T19:11:24.1145559Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:11:24.1145819Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:11:24.1146070Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:11:24.1146491Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:11:24.1146981Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:11:24.1147285Z TEST_CONFIG: distributed 2022-03-17T19:11:24.1147503Z SHARD_NUMBER: 1 2022-03-17T19:11:24.1147721Z NUM_TEST_SHARDS: 1 2022-03-17T19:11:24.1147927Z PR_BODY: 2022-03-17T19:11:24.1148185Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:11:24.1148534Z ##[endgroup] 2022-03-17T19:11:25.3110112Z c3998772049b8a8ba52cedfa525c9feda04cb2c7: Pulling from pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:11:25.3111618Z cf06a7c31611: Pulling fs layer 2022-03-17T19:11:25.3112591Z fe85c9f573f1: Pulling fs layer 2022-03-17T19:11:25.3113541Z 2c9c5600e0e9: Pulling fs layer 2022-03-17T19:11:25.3114518Z cfae2268fadf: Pulling fs layer 2022-03-17T19:11:25.3115465Z 988d70811ea2: Pulling fs layer 2022-03-17T19:11:25.3116394Z 39d7108477b9: Pulling fs layer 2022-03-17T19:11:25.3117252Z 7301acfbb12d: Pulling fs layer 2022-03-17T19:11:25.3118075Z 71ab17f490c8: Pulling fs layer 2022-03-17T19:11:25.3118939Z e5c4d1ff74c9: Pulling fs layer 2022-03-17T19:11:25.3119657Z 6ae328336172: Pulling fs layer 2022-03-17T19:11:25.3120277Z c36d498385de: Pulling fs layer 2022-03-17T19:11:25.3120882Z 827f6f3d5730: Pulling fs layer 2022-03-17T19:11:25.3121703Z a6e2aafb2124: Pulling fs layer 2022-03-17T19:11:25.3123162Z 2f5b5552b80b: Pulling fs layer 2022-03-17T19:11:25.3124188Z a1a519874d1e: Pulling fs layer 2022-03-17T19:11:25.3125027Z 0d6fb4e09e2a: Pulling fs layer 2022-03-17T19:11:25.3125848Z 0c1367a60cae: Pulling fs layer 2022-03-17T19:11:25.3126805Z 8bc49599431a: Pulling fs layer 2022-03-17T19:11:25.3127608Z 62811a489ca4: Pulling fs layer 2022-03-17T19:11:25.3128494Z afd99156b1ea: Pulling fs layer 2022-03-17T19:11:25.3129141Z a8dc5e4a749a: Pulling fs layer 2022-03-17T19:11:25.3129772Z 925686690363: Pulling fs layer 2022-03-17T19:11:25.3130400Z 6ddd61a840be: Pulling fs layer 2022-03-17T19:11:25.3131043Z bb1d57859fc8: Pulling fs layer 2022-03-17T19:11:25.3131682Z bae8cb409177: Pulling fs layer 2022-03-17T19:11:25.3132310Z 27ae01b3f66b: Pulling fs layer 2022-03-17T19:11:25.3132952Z dd4766f67bd3: Pulling fs layer 2022-03-17T19:11:25.3133548Z a6e2aafb2124: Waiting 2022-03-17T19:11:25.3134162Z ffbd4bc79c6d: Pulling fs layer 2022-03-17T19:11:25.3134765Z 2f5b5552b80b: Waiting 2022-03-17T19:11:25.3135915Z 66de067bcf31: Pulling fs layer 2022-03-17T19:11:25.3136535Z c36d498385de: Waiting 2022-03-17T19:11:25.3137076Z a1a519874d1e: Waiting 2022-03-17T19:11:25.3137631Z 988d70811ea2: Waiting 2022-03-17T19:11:25.3138157Z 827f6f3d5730: Waiting 2022-03-17T19:11:25.3138749Z cfae2268fadf: Waiting 2022-03-17T19:11:25.3139324Z 7301acfbb12d: Waiting 2022-03-17T19:11:25.3139859Z 39d7108477b9: Waiting 2022-03-17T19:11:25.3140398Z 71ab17f490c8: Waiting 2022-03-17T19:11:25.3140941Z e5c4d1ff74c9: Waiting 2022-03-17T19:11:25.3141494Z 8bc49599431a: Waiting 2022-03-17T19:11:25.3142031Z 6ddd61a840be: Waiting 2022-03-17T19:11:25.3142609Z ffbd4bc79c6d: Waiting 2022-03-17T19:11:25.3143166Z 6ae328336172: Waiting 2022-03-17T19:11:25.3143697Z 0d6fb4e09e2a: Waiting 2022-03-17T19:11:25.3144246Z 62811a489ca4: Waiting 2022-03-17T19:11:25.3144776Z bae8cb409177: Waiting 2022-03-17T19:11:25.3145323Z 66de067bcf31: Waiting 2022-03-17T19:11:25.3145861Z bb1d57859fc8: Waiting 2022-03-17T19:11:25.3146409Z afd99156b1ea: Waiting 2022-03-17T19:11:25.3146939Z 925686690363: Waiting 2022-03-17T19:11:25.3147475Z 27ae01b3f66b: Waiting 2022-03-17T19:11:25.3148040Z a8dc5e4a749a: Waiting 2022-03-17T19:11:25.9732696Z fe85c9f573f1: Verifying Checksum 2022-03-17T19:11:25.9733438Z fe85c9f573f1: Download complete 2022-03-17T19:11:26.6376233Z cfae2268fadf: Download complete 2022-03-17T19:11:27.2952055Z 988d70811ea2: Download complete 2022-03-17T19:11:27.9485615Z 39d7108477b9: Verifying Checksum 2022-03-17T19:11:27.9486533Z 39d7108477b9: Download complete 2022-03-17T19:11:28.5923847Z 7301acfbb12d: Verifying Checksum 2022-03-17T19:11:28.5924923Z 7301acfbb12d: Download complete 2022-03-17T19:11:29.2364843Z 71ab17f490c8: Verifying Checksum 2022-03-17T19:11:29.2365810Z 71ab17f490c8: Download complete 2022-03-17T19:11:32.4212665Z 2c9c5600e0e9: Verifying Checksum 2022-03-17T19:11:32.4216014Z 2c9c5600e0e9: Download complete 2022-03-17T19:11:33.1700055Z 6ae328336172: Verifying Checksum 2022-03-17T19:11:41.3816877Z c36d498385de: Verifying Checksum 2022-03-17T19:11:41.3817653Z c36d498385de: Download complete 2022-03-17T19:11:42.2167363Z 827f6f3d5730: Verifying Checksum 2022-03-17T19:11:42.2168520Z 827f6f3d5730: Download complete 2022-03-17T19:11:43.4822949Z cf06a7c31611: Verifying Checksum 2022-03-17T19:11:43.4823858Z cf06a7c31611: Download complete 2022-03-17T19:11:44.3444121Z 2f5b5552b80b: Verifying Checksum 2022-03-17T19:11:44.3444666Z 2f5b5552b80b: Download complete 2022-03-17T19:11:44.6278065Z cf06a7c31611: Pull complete 2022-03-17T19:11:45.0981177Z fe85c9f573f1: Pull complete 2022-03-17T19:11:45.2166735Z a1a519874d1e: Verifying Checksum 2022-03-17T19:11:45.2167432Z a1a519874d1e: Download complete 2022-03-17T19:11:46.3849711Z 0d6fb4e09e2a: Verifying Checksum 2022-03-17T19:11:46.3850661Z 0d6fb4e09e2a: Download complete 2022-03-17T19:11:47.1148668Z 0c1367a60cae: Download complete 2022-03-17T19:11:47.7909051Z 8bc49599431a: Download complete 2022-03-17T19:11:55.5374586Z 2c9c5600e0e9: Pull complete 2022-03-17T19:11:55.5999591Z cfae2268fadf: Pull complete 2022-03-17T19:11:55.6599644Z 988d70811ea2: Pull complete 2022-03-17T19:11:55.7229017Z 39d7108477b9: Pull complete 2022-03-17T19:11:55.7835213Z 7301acfbb12d: Pull complete 2022-03-17T19:11:55.8473528Z 71ab17f490c8: Pull complete 2022-03-17T19:12:11.6390928Z a6e2aafb2124: Verifying Checksum 2022-03-17T19:12:11.6392021Z a6e2aafb2124: Download complete 2022-03-17T19:12:12.5713175Z afd99156b1ea: Download complete 2022-03-17T19:12:13.2950509Z a8dc5e4a749a: Verifying Checksum 2022-03-17T19:12:13.2951562Z a8dc5e4a749a: Download complete 2022-03-17T19:13:07.8663025Z 62811a489ca4: Verifying Checksum 2022-03-17T19:13:07.8663977Z 62811a489ca4: Download complete 2022-03-17T19:13:08.9122624Z 6ddd61a840be: Download complete 2022-03-17T19:13:09.7434191Z bb1d57859fc8: Verifying Checksum 2022-03-17T19:13:09.7435091Z bb1d57859fc8: Download complete 2022-03-17T19:13:10.6126586Z bae8cb409177: Verifying Checksum 2022-03-17T19:13:10.6127615Z bae8cb409177: Download complete 2022-03-17T19:13:11.4933921Z 27ae01b3f66b: Verifying Checksum 2022-03-17T19:13:11.4935118Z 27ae01b3f66b: Download complete 2022-03-17T19:13:12.3976158Z dd4766f67bd3: Download complete 2022-03-17T19:13:13.2490251Z ffbd4bc79c6d: Verifying Checksum 2022-03-17T19:13:13.2491014Z ffbd4bc79c6d: Download complete 2022-03-17T19:13:15.4504317Z 66de067bcf31: Verifying Checksum 2022-03-17T19:13:15.4505137Z 66de067bcf31: Download complete 2022-03-17T19:15:52.4255761Z e5c4d1ff74c9: Verifying Checksum 2022-03-17T19:15:52.4260732Z e5c4d1ff74c9: Download complete 2022-03-17T19:16:27.8674101Z e5c4d1ff74c9: Pull complete 2022-03-17T19:16:27.9289502Z 6ae328336172: Pull complete 2022-03-17T19:16:29.4905896Z c36d498385de: Pull complete 2022-03-17T19:16:29.5593592Z 827f6f3d5730: Pull complete 2022-03-17T19:16:35.3453085Z a6e2aafb2124: Pull complete 2022-03-17T19:16:35.4181788Z 2f5b5552b80b: Pull complete 2022-03-17T19:16:35.4804636Z a1a519874d1e: Pull complete 2022-03-17T19:16:35.6029714Z 0d6fb4e09e2a: Pull complete 2022-03-17T19:16:35.6700776Z 0c1367a60cae: Pull complete 2022-03-17T19:16:35.7309830Z 8bc49599431a: Pull complete 2022-03-17T19:16:44.9661517Z 62811a489ca4: Pull complete 2022-03-17T19:16:45.0330517Z afd99156b1ea: Pull complete 2022-03-17T19:16:45.0908495Z a8dc5e4a749a: Pull complete 2022-03-17T19:21:26.1157451Z 925686690363: Verifying Checksum 2022-03-17T19:21:26.1158563Z 925686690363: Download complete 2022-03-17T19:23:37.6555282Z 925686690363: Pull complete 2022-03-17T19:23:37.7237000Z 6ddd61a840be: Pull complete 2022-03-17T19:23:37.7937296Z bb1d57859fc8: Pull complete 2022-03-17T19:23:37.8571634Z bae8cb409177: Pull complete 2022-03-17T19:23:37.9296562Z 27ae01b3f66b: Pull complete 2022-03-17T19:23:37.9899372Z dd4766f67bd3: Pull complete 2022-03-17T19:23:38.0520957Z ffbd4bc79c6d: Pull complete 2022-03-17T19:23:38.7685579Z 66de067bcf31: Pull complete 2022-03-17T19:23:38.7727676Z Digest: sha256:a482b3a3008d87393da82f727e1eccc5f8ab041e50908ba7de84e9561c04f097 2022-03-17T19:23:38.7736169Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:23:38.7757235Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:23:38.7849445Z ##[group]Run echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-03-17T19:23:38.7851003Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-03-17T19:23:38.7909039Z shell: /bin/bash -e {0} 2022-03-17T19:23:38.7909263Z env: 2022-03-17T19:23:38.7909543Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:23:38.7909997Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:23:38.7910427Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:23:38.7910799Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:23:38.7911126Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:23:38.7911341Z IN_CI: 1 2022-03-17T19:23:38.7911534Z IS_GHA: 1 2022-03-17T19:23:38.7911739Z IN_WHEEL_TEST: 1 2022-03-17T19:23:38.7912031Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:23:38.7912375Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:23:38.7912666Z PR_LABELS: [] 2022-03-17T19:23:38.7913090Z GITHUB_TOKEN: *** 2022-03-17T19:23:38.7913321Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:23:38.7913554Z PR_NUMBER: 2022-03-17T19:23:38.7913788Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:23:38.7914046Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:23:38.7914299Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:23:38.7914717Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:23:38.7915202Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:23:38.7915506Z TEST_CONFIG: distributed 2022-03-17T19:23:38.7915731Z SHARD_NUMBER: 1 2022-03-17T19:23:38.7915950Z NUM_TEST_SHARDS: 1 2022-03-17T19:23:38.7916159Z PR_BODY: 2022-03-17T19:23:38.7916413Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:23:38.7916729Z ##[endgroup] 2022-03-17T19:23:38.8047188Z ##[group]Run shm_size="1g" 2022-03-17T19:23:38.8047875Z shm_size="1g" 2022-03-17T19:23:38.8048523Z case "${BUILD_ENVIRONMENT}" in 2022-03-17T19:23:38.8049127Z  *cuda*) 2022-03-17T19:23:38.8049686Z  shm_size="2g" 2022-03-17T19:23:38.8050226Z  ;; 2022-03-17T19:23:38.8050751Z  *rocm*) 2022-03-17T19:23:38.8051293Z  shm_size="8g" 2022-03-17T19:23:38.8051842Z  ;; 2022-03-17T19:23:38.8052361Z esac 2022-03-17T19:23:38.8052994Z echo "SHM_SIZE=${shm_size}" >> "${GITHUB_ENV}" 2022-03-17T19:23:38.8106887Z shell: /bin/bash -e {0} 2022-03-17T19:23:38.8107099Z env: 2022-03-17T19:23:38.8107370Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:23:38.8107820Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:23:38.8108245Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:23:38.8108644Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:23:38.8108964Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:23:38.8109186Z IN_CI: 1 2022-03-17T19:23:38.8109384Z IS_GHA: 1 2022-03-17T19:23:38.8109581Z IN_WHEEL_TEST: 1 2022-03-17T19:23:38.8109858Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:23:38.8110202Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:23:38.8110495Z PR_LABELS: [] 2022-03-17T19:23:38.8110949Z GITHUB_TOKEN: *** 2022-03-17T19:23:38.8111179Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:23:38.8111415Z PR_NUMBER: 2022-03-17T19:23:38.8111648Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:23:38.8112012Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:23:38.8112262Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:23:38.8112694Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:23:38.8113186Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:23:38.8113479Z TEST_CONFIG: distributed 2022-03-17T19:23:38.8113708Z SHARD_NUMBER: 1 2022-03-17T19:23:38.8113926Z NUM_TEST_SHARDS: 1 2022-03-17T19:23:38.8114131Z PR_BODY: 2022-03-17T19:23:38.8114383Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:23:38.8114753Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T19:23:38.8115084Z ##[endgroup] 2022-03-17T19:23:38.8236254Z ##[group]Run seemethere/download-artifact-s3@0504774707cbc8603d7dca922e8026eb8bf3b47b 2022-03-17T19:23:38.8236817Z with: 2022-03-17T19:23:38.8237305Z name: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:23:38.8237834Z s3-bucket: gha-artifacts 2022-03-17T19:23:38.8238273Z region: us-east-1 2022-03-17T19:23:38.8238645Z env: 2022-03-17T19:23:38.8239155Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:23:38.8239994Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:23:38.8240780Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:23:38.8241497Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:23:38.8242351Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:23:38.8242743Z IN_CI: 1 2022-03-17T19:23:38.8243102Z IS_GHA: 1 2022-03-17T19:23:38.8243472Z IN_WHEEL_TEST: 1 2022-03-17T19:23:38.8244028Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:23:38.8244687Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:23:38.8245230Z PR_LABELS: [] 2022-03-17T19:23:38.8245959Z GITHUB_TOKEN: *** 2022-03-17T19:23:38.8246394Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:23:38.8246831Z PR_NUMBER: 2022-03-17T19:23:38.8247273Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:23:38.8247752Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:23:38.8248213Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:23:38.8249006Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:23:38.8249924Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:23:38.8250485Z TEST_CONFIG: distributed 2022-03-17T19:23:38.8250903Z SHARD_NUMBER: 1 2022-03-17T19:23:38.8251292Z NUM_TEST_SHARDS: 1 2022-03-17T19:23:38.8251688Z PR_BODY: 2022-03-17T19:23:38.8252157Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:23:38.8252857Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T19:23:38.8253470Z SHM_SIZE: 8g 2022-03-17T19:23:38.8253848Z ##[endgroup] 2022-03-17T19:23:39.7199958Z Found 1 objects with prefix pytorch/pytorch/2000264370/linux-bionic-rocm4.5-py3.7-distributed/ 2022-03-17T19:23:39.7201545Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-03-17T19:25:40.6664998Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-03-17T19:25:40.6666574Z 2022-03-17T19:25:40.6680890Z Artifact download has finished successfully 2022-03-17T19:25:40.6817259Z ##[group]Run unzip -o artifacts.zip 2022-03-17T19:25:40.6818060Z unzip -o artifacts.zip 2022-03-17T19:25:40.6895797Z shell: /bin/bash -e {0} 2022-03-17T19:25:40.6896010Z env: 2022-03-17T19:25:40.6896294Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:25:40.6896736Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:25:40.6897167Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:25:40.6897647Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:25:40.6897973Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:25:40.6898194Z IN_CI: 1 2022-03-17T19:25:40.6898382Z IS_GHA: 1 2022-03-17T19:25:40.6898586Z IN_WHEEL_TEST: 1 2022-03-17T19:25:40.6898857Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:25:40.6899208Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:25:40.6899514Z PR_LABELS: [] 2022-03-17T19:25:40.6899995Z GITHUB_TOKEN: *** 2022-03-17T19:25:40.6900222Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:25:40.6900452Z PR_NUMBER: 2022-03-17T19:25:40.6900683Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:25:40.6900936Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:25:40.6901175Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:25:40.6901603Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:25:40.6902092Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:25:40.6902387Z TEST_CONFIG: distributed 2022-03-17T19:25:40.6902610Z SHARD_NUMBER: 1 2022-03-17T19:25:40.6902819Z NUM_TEST_SHARDS: 1 2022-03-17T19:25:40.6903033Z PR_BODY: 2022-03-17T19:25:40.6903281Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:25:40.6903651Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T19:25:40.6903978Z SHM_SIZE: 8g 2022-03-17T19:25:40.6904177Z ##[endgroup] 2022-03-17T19:25:40.6970219Z Archive: artifacts.zip 2022-03-17T19:25:40.6971937Z creating: dist/ 2022-03-17T19:25:42.1259856Z inflating: dist/torch-1.12.0a0+git4b4f652-cp37-cp37m-linux_x86_64.whl 2022-03-17T19:25:42.1260736Z creating: build/lib/ 2022-03-17T19:25:42.1261377Z inflating: build/lib/libclog.a 2022-03-17T19:25:42.1329126Z inflating: build/lib/libgtest.a 2022-03-17T19:25:42.1338998Z inflating: build/lib/libpthreadpool.a 2022-03-17T19:25:42.1416660Z inflating: build/lib/libbenchmark.a 2022-03-17T19:25:42.1509988Z inflating: build/lib/libprotobuf-lite.a 2022-03-17T19:25:42.1560203Z inflating: build/lib/libasmjit.a 2022-03-17T19:25:42.1590162Z inflating: build/lib/libtensorpipe_uv.a 2022-03-17T19:25:42.1690496Z inflating: build/lib/libgloo.a 2022-03-17T19:25:42.1709762Z inflating: build/lib/libfmt.a 2022-03-17T19:25:42.2180066Z inflating: build/lib/libprotobuf.a 2022-03-17T19:25:42.2184912Z inflating: build/lib/libbreakpad_common.a 2022-03-17T19:25:42.2186202Z inflating: build/lib/libcaffe2_nvrtc.so 2022-03-17T19:25:42.2187108Z inflating: build/lib/libfoxi_loader.a 2022-03-17T19:25:42.2237494Z inflating: build/lib/libc10.so 2022-03-17T19:25:42.2238536Z inflating: build/lib/libtorch_global_deps.so 2022-03-17T19:25:42.2247156Z inflating: build/lib/libcpuinfo.a 2022-03-17T19:25:42.2248769Z inflating: build/lib/libnnpack_reference_layers.a 2022-03-17T19:25:42.2259064Z inflating: build/lib/libcpuinfo_internals.a 2022-03-17T19:25:42.2782773Z inflating: build/lib/libprotoc.a 2022-03-17T19:25:42.2803039Z inflating: build/lib/libgmock.a 2022-03-17T19:25:42.2804062Z inflating: build/lib/libbenchmark_main.a 2022-03-17T19:25:42.2804855Z inflating: build/lib/libgtest_main.a 2022-03-17T19:25:43.0029656Z inflating: build/lib/libdnnl.a 2022-03-17T19:25:43.0435290Z inflating: build/lib/libgloo_hip.a 2022-03-17T19:25:43.0452322Z inflating: build/lib/libbreakpad.a 2022-03-17T19:25:43.1046731Z inflating: build/lib/libtensorpipe.a 2022-03-17T19:25:43.1082216Z inflating: build/lib/libc10_hip.so 2022-03-17T19:25:43.1098483Z inflating: build/lib/libqnnpack.a 2022-03-17T19:25:43.1218647Z inflating: build/lib/libXNNPACK.a 2022-03-17T19:25:43.1219678Z inflating: build/lib/libgmock_main.a 2022-03-17T19:25:43.2369899Z inflating: build/lib/libfbgemm.a 2022-03-17T19:25:43.2631198Z inflating: build/lib/libkineto.a 2022-03-17T19:25:43.2654806Z inflating: build/lib/libpytorch_qnnpack.a 2022-03-17T19:25:43.2697762Z inflating: build/lib/libcaffe2_protos.a 2022-03-17T19:25:43.2742291Z inflating: build/lib/libonnx_proto.a 2022-03-17T19:25:43.3328490Z inflating: build/lib/libonnx.a 2022-03-17T19:25:43.3349026Z inflating: build/lib/libnnpack.a 2022-03-17T19:25:44.7912471Z inflating: build/lib/libtorch_cpu.so 2022-03-17T19:25:47.9211069Z inflating: build/lib/libtorch_hip.so 2022-03-17T19:25:47.9212733Z inflating: build/lib/libtorch.so 2022-03-17T19:25:47.9235875Z inflating: build/lib/libjitbackend_test.so 2022-03-17T19:25:47.9288306Z inflating: build/lib/libtorchbind_test.so 2022-03-17T19:25:47.9316481Z inflating: build/lib/libbackend_with_compiler.so 2022-03-17T19:25:47.9319750Z inflating: build/lib/libshm.so 2022-03-17T19:25:48.0735508Z inflating: build/lib/libtorch_python.so 2022-03-17T19:25:48.0772128Z inflating: build/lib/libnnapi_backend.so 2022-03-17T19:25:48.0773117Z creating: build/bin/ 2022-03-17T19:25:48.0824798Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-03-17T19:25:48.0879537Z inflating: build/bin/c10_DeviceGuard_test 2022-03-17T19:25:48.0932424Z inflating: build/bin/c10_Device_test 2022-03-17T19:25:48.0988666Z inflating: build/bin/c10_DispatchKeySet_test 2022-03-17T19:25:48.1039954Z inflating: build/bin/c10_StreamGuard_test 2022-03-17T19:25:48.1096876Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-03-17T19:25:48.1155918Z inflating: build/bin/c10_InlineStreamGuard_test 2022-03-17T19:25:48.1215884Z inflating: build/bin/c10_SizesAndStrides_test 2022-03-17T19:25:48.1267822Z inflating: build/bin/c10_Array_test 2022-03-17T19:25:48.1323740Z inflating: build/bin/c10_Bitset_test 2022-03-17T19:25:48.1374474Z inflating: build/bin/c10_ConstexprCrc_test 2022-03-17T19:25:48.1429626Z inflating: build/bin/c10_C++17_test 2022-03-17T19:25:48.1481761Z inflating: build/bin/c10_Half_test 2022-03-17T19:25:48.1540676Z inflating: build/bin/c10_LeftRight_test 2022-03-17T19:25:48.1607720Z inflating: build/bin/c10_Metaprogramming_test 2022-03-17T19:25:48.1765986Z inflating: build/bin/c10_SmallVectorTest 2022-03-17T19:25:48.1819346Z inflating: build/bin/c10_Synchronized_test 2022-03-17T19:25:48.1877907Z inflating: build/bin/c10_ThreadLocal_test 2022-03-17T19:25:48.1932448Z inflating: build/bin/c10_TypeIndex_test 2022-03-17T19:25:48.1985849Z inflating: build/bin/c10_TypeList_test 2022-03-17T19:25:48.2036240Z inflating: build/bin/c10_TypeTraits_test 2022-03-17T19:25:48.2091501Z inflating: build/bin/c10_accumulate_test 2022-03-17T19:25:48.2150696Z inflating: build/bin/c10_bfloat16_test 2022-03-17T19:25:48.2209798Z inflating: build/bin/c10_complex_math_test 2022-03-17T19:25:48.2267171Z inflating: build/bin/c10_complex_test 2022-03-17T19:25:48.2320825Z inflating: build/bin/c10_flags_test 2022-03-17T19:25:48.2376338Z inflating: build/bin/c10_exception_test 2022-03-17T19:25:48.2490886Z inflating: build/bin/c10_either_test 2022-03-17T19:25:48.2544412Z inflating: build/bin/c10_irange_test 2022-03-17T19:25:48.2602537Z inflating: build/bin/c10_logging_test 2022-03-17T19:25:48.2773024Z inflating: build/bin/c10_intrusive_ptr_test 2022-03-17T19:25:48.2851071Z inflating: build/bin/c10_optional_test 2022-03-17T19:25:48.2916683Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-03-17T19:25:48.2974590Z inflating: build/bin/c10_registry_test 2022-03-17T19:25:48.3037706Z inflating: build/bin/c10_string_view_test 2022-03-17T19:25:48.3092281Z inflating: build/bin/c10_tempfile_test 2022-03-17T19:25:48.3143449Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-03-17T19:25:48.3201619Z inflating: build/bin/c10_typeid_test 2022-03-17T19:25:48.3664783Z inflating: build/bin/protoc-3.13.0.0 2022-03-17T19:25:48.4126712Z inflating: build/bin/protoc 2022-03-17T19:25:48.4178095Z inflating: build/bin/c10_hip_HIPTest 2022-03-17T19:25:48.4477385Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-03-17T19:25:48.4799291Z inflating: build/bin/vec_test_all_types_AVX512 2022-03-17T19:25:48.5135518Z inflating: build/bin/vec_test_all_types_AVX2 2022-03-17T19:25:48.5191118Z inflating: build/bin/FileStoreTest 2022-03-17T19:25:48.5255702Z inflating: build/bin/TCPStoreTest 2022-03-17T19:25:48.5311816Z inflating: build/bin/HashStoreTest 2022-03-17T19:25:48.5380167Z inflating: build/bin/ProcessGroupGlooTest 2022-03-17T19:25:48.5381341Z inflating: build/bin/example_allreduce 2022-03-17T19:25:48.5436563Z inflating: build/bin/Dimname_test 2022-03-17T19:25:48.5496299Z inflating: build/bin/scalar_test 2022-03-17T19:25:48.5558918Z inflating: build/bin/apply_utils_test 2022-03-17T19:25:48.5639192Z inflating: build/bin/Dict_test 2022-03-17T19:25:48.5702879Z inflating: build/bin/basic 2022-03-17T19:25:48.5765315Z inflating: build/bin/atest 2022-03-17T19:25:48.5825983Z inflating: build/bin/NamedTensor_test 2022-03-17T19:25:48.5883904Z inflating: build/bin/half_test 2022-03-17T19:25:48.5938558Z inflating: build/bin/wrapdim_test 2022-03-17T19:25:48.5995096Z inflating: build/bin/broadcast_test 2022-03-17T19:25:48.6054452Z inflating: build/bin/native_test 2022-03-17T19:25:48.6107821Z inflating: build/bin/dlconvertor_test 2022-03-17T19:25:48.6167061Z inflating: build/bin/scalar_tensor_test 2022-03-17T19:25:48.6221919Z inflating: build/bin/undefined_tensor_test 2022-03-17T19:25:48.6223000Z inflating: build/bin/verify_api_visibility 2022-03-17T19:25:48.6224094Z inflating: build/bin/thread_init_test 2022-03-17T19:25:48.6281496Z inflating: build/bin/test_parallel 2022-03-17T19:25:48.6334947Z inflating: build/bin/weakref_test 2022-03-17T19:25:48.6395141Z inflating: build/bin/quantized_test 2022-03-17T19:25:48.6455053Z inflating: build/bin/extension_backend_test 2022-03-17T19:25:48.6509315Z inflating: build/bin/operators_test 2022-03-17T19:25:48.6561428Z inflating: build/bin/lazy_tensor_test 2022-03-17T19:25:48.6641780Z inflating: build/bin/tensor_iterator_test 2022-03-17T19:25:48.6699054Z inflating: build/bin/math_kernel_test 2022-03-17T19:25:48.6753958Z inflating: build/bin/memory_overlapping_test 2022-03-17T19:25:48.6814917Z inflating: build/bin/cpu_generator_test 2022-03-17T19:25:48.6866292Z inflating: build/bin/variant_test 2022-03-17T19:25:48.6922871Z inflating: build/bin/cpu_profiling_allocator_test 2022-03-17T19:25:48.6977867Z inflating: build/bin/mobile_memory_cleanup 2022-03-17T19:25:48.7046195Z inflating: build/bin/pow_test 2022-03-17T19:25:48.7099354Z inflating: build/bin/reduce_ops_test 2022-03-17T19:25:48.7153269Z inflating: build/bin/reportMemoryUsage_test 2022-03-17T19:25:48.7208211Z inflating: build/bin/memory_format_test 2022-03-17T19:25:48.7300428Z inflating: build/bin/cpu_rng_test 2022-03-17T19:25:48.7372578Z inflating: build/bin/vmap_test 2022-03-17T19:25:48.7470270Z inflating: build/bin/ivalue_test 2022-03-17T19:25:48.7533669Z inflating: build/bin/type_test 2022-03-17T19:25:48.7585489Z inflating: build/bin/dispatch_key_set_test 2022-03-17T19:25:48.7696915Z inflating: build/bin/List_test 2022-03-17T19:25:48.7751505Z inflating: build/bin/stride_properties_test 2022-03-17T19:25:48.7873207Z inflating: build/bin/kernel_function_legacy_test 2022-03-17T19:25:48.7940758Z inflating: build/bin/KernelFunction_test 2022-03-17T19:25:48.8039983Z inflating: build/bin/kernel_function_test 2022-03-17T19:25:48.8168723Z inflating: build/bin/kernel_lambda_legacy_test 2022-03-17T19:25:48.8231929Z inflating: build/bin/kernel_stackbased_test 2022-03-17T19:25:48.8339510Z inflating: build/bin/kernel_lambda_test 2022-03-17T19:25:48.8438614Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-03-17T19:25:48.8493344Z inflating: build/bin/CppSignature_test 2022-03-17T19:25:48.8544449Z inflating: build/bin/op_allowlist_test 2022-03-17T19:25:48.8595773Z inflating: build/bin/hip_complex_test 2022-03-17T19:25:48.8652849Z inflating: build/bin/backend_fallback_test 2022-03-17T19:25:48.8707925Z inflating: build/bin/inline_container_test 2022-03-17T19:25:48.8758708Z inflating: build/bin/hip_complex_math_test 2022-03-17T19:25:48.9056904Z inflating: build/bin/op_registration_test 2022-03-17T19:25:48.9108222Z inflating: build/bin/hip_integer_divider_test 2022-03-17T19:25:48.9160805Z inflating: build/bin/hip_half_test 2022-03-17T19:25:48.9213010Z inflating: build/bin/hip_distributions_test 2022-03-17T19:25:48.9264839Z inflating: build/bin/hip_optional_test 2022-03-17T19:25:48.9319893Z inflating: build/bin/hip_apply_test 2022-03-17T19:25:48.9372651Z inflating: build/bin/hip_packedtensoraccessor_test 2022-03-17T19:25:48.9424376Z inflating: build/bin/hip_vectorized_test 2022-03-17T19:25:48.9475989Z inflating: build/bin/hip_generator_test 2022-03-17T19:25:48.9528637Z inflating: build/bin/hip_dlconvertor_test 2022-03-17T19:25:48.9543120Z inflating: build/bin/tutorial_tensorexpr 2022-03-17T19:25:48.9600439Z inflating: build/bin/test_dist_autograd 2022-03-17T19:25:48.9601706Z inflating: build/bin/parallel_benchmark 2022-03-17T19:25:48.9675833Z inflating: build/bin/test_cpp_rpc 2022-03-17T19:25:48.9764451Z inflating: build/bin/test_lazy 2022-03-17T19:25:48.9768868Z inflating: build/bin/torch_shm_manager 2022-03-17T19:25:49.0465508Z inflating: build/bin/test_tensorexpr 2022-03-17T19:25:49.1596877Z inflating: build/bin/test_api 2022-03-17T19:25:49.2113578Z inflating: build/bin/test_jit 2022-03-17T19:25:49.2172290Z ##[group]Run df -H 2022-03-17T19:25:49.2172914Z df -H 2022-03-17T19:25:49.2226176Z shell: /bin/bash -e {0} 2022-03-17T19:25:49.2226395Z env: 2022-03-17T19:25:49.2226676Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:25:49.2227124Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:25:49.2227564Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:25:49.2227938Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:25:49.2228268Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:25:49.2228479Z IN_CI: 1 2022-03-17T19:25:49.2228678Z IS_GHA: 1 2022-03-17T19:25:49.2228886Z IN_WHEEL_TEST: 1 2022-03-17T19:25:49.2229156Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:25:49.2229504Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:25:49.2229787Z PR_LABELS: [] 2022-03-17T19:25:49.2230590Z GITHUB_TOKEN: *** 2022-03-17T19:25:49.2230830Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:25:49.2231064Z PR_NUMBER: 2022-03-17T19:25:49.2231303Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:25:49.2231557Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:25:49.2231814Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:25:49.2232238Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:25:49.2232732Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:25:49.2233047Z TEST_CONFIG: distributed 2022-03-17T19:25:49.2233268Z SHARD_NUMBER: 1 2022-03-17T19:25:49.2233484Z NUM_TEST_SHARDS: 1 2022-03-17T19:25:49.2233692Z PR_BODY: 2022-03-17T19:25:49.2233944Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:25:49.2234321Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T19:25:49.2234655Z SHM_SIZE: 8g 2022-03-17T19:25:49.2234862Z ##[endgroup] 2022-03-17T19:25:49.2316403Z Filesystem Size Used Avail Use% Mounted on 2022-03-17T19:25:49.2317507Z udev 136G 0 136G 0% /dev 2022-03-17T19:25:49.2318543Z tmpfs 28G 1.5M 28G 1% /run 2022-03-17T19:25:49.2319863Z /dev/nvme0n1p1 984G 337G 597G 37% / 2022-03-17T19:25:49.2320694Z tmpfs 136G 8.2k 136G 1% /dev/shm 2022-03-17T19:25:49.2324178Z tmpfs 5.3M 0 5.3M 0% /run/lock 2022-03-17T19:25:49.2325050Z tmpfs 136G 0 136G 0% /sys/fs/cgroup 2022-03-17T19:25:49.2325764Z tmpfs 28G 13k 28G 1% /run/user/1123 2022-03-17T19:25:49.2367221Z ##[group]Run ./.github/scripts/parse_ref.py 2022-03-17T19:25:49.2368061Z ./.github/scripts/parse_ref.py 2022-03-17T19:25:49.2452914Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T19:25:49.2453629Z env: 2022-03-17T19:25:49.2454379Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:25:49.2455009Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:25:49.2455489Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:25:49.2455862Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:25:49.2456197Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:25:49.2456414Z IN_CI: 1 2022-03-17T19:25:49.2456603Z IS_GHA: 1 2022-03-17T19:25:49.2456810Z IN_WHEEL_TEST: 1 2022-03-17T19:25:49.2457082Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:25:49.2457451Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:25:49.2457733Z PR_LABELS: [] 2022-03-17T19:25:49.2458175Z GITHUB_TOKEN: *** 2022-03-17T19:25:49.2458420Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:25:49.2458648Z PR_NUMBER: 2022-03-17T19:25:49.2458894Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:25:49.2459147Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:25:49.2459398Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:25:49.2459824Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:25:49.2460324Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:25:49.2460633Z TEST_CONFIG: distributed 2022-03-17T19:25:49.2460855Z SHARD_NUMBER: 1 2022-03-17T19:25:49.2461073Z NUM_TEST_SHARDS: 1 2022-03-17T19:25:49.2461280Z PR_BODY: 2022-03-17T19:25:49.2461536Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:25:49.2461921Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T19:25:49.2462246Z SHM_SIZE: 8g 2022-03-17T19:25:49.2462454Z ##[endgroup] 2022-03-17T19:25:49.2904172Z ##[group]Run set -x 2022-03-17T19:25:49.2905057Z set -x 2022-03-17T19:25:49.2905602Z  2022-03-17T19:25:49.2906281Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-03-17T19:25:49.2907175Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-03-17T19:25:49.2908050Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-03-17T19:25:49.2908872Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-03-17T19:25:49.2909565Z else 2022-03-17T19:25:49.2910272Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-03-17T19:25:49.2910951Z fi 2022-03-17T19:25:49.2911739Z # detached container should get cleaned up by teardown_ec2_linux 2022-03-17T19:25:49.2912772Z # TODO: Stop building test binaries as part of the build phase 2022-03-17T19:25:49.2913716Z # Used for GPU_FLAG since that doesn't play nice 2022-03-17T19:25:49.2914552Z # shellcheck disable=SC2086,SC2090 2022-03-17T19:25:49.2915351Z container_name=$(docker run \ 2022-03-17T19:25:49.2916029Z  ${GPU_FLAG:-} \ 2022-03-17T19:25:49.2916710Z  -e BUILD_ENVIRONMENT \ 2022-03-17T19:25:49.2917396Z  -e PR_NUMBER \ 2022-03-17T19:25:49.2918127Z  -e CUSTOM_TEST_ARTIFACT_BUILD_DIR \ 2022-03-17T19:25:49.2918861Z  -e GITHUB_ACTIONS \ 2022-03-17T19:25:49.2919488Z  -e IN_CI \ 2022-03-17T19:25:49.2920084Z  -e IS_GHA \ 2022-03-17T19:25:49.2920900Z  -e BRANCH \ 2022-03-17T19:25:49.2921503Z  -e SHA1 \ 2022-03-17T19:25:49.2922296Z  -e AWS_DEFAULT_REGION \ 2022-03-17T19:25:49.2922963Z  -e IN_WHEEL_TEST \ 2022-03-17T19:25:49.2923616Z  -e SHARD_NUMBER \ 2022-03-17T19:25:49.2924261Z  -e JOB_BASE_NAME \ 2022-03-17T19:25:49.2924910Z  -e TEST_CONFIG \ 2022-03-17T19:25:49.2925572Z  -e NUM_TEST_SHARDS \ 2022-03-17T19:25:49.2926207Z  -e PR_BODY \ 2022-03-17T19:25:49.2926920Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-03-17T19:25:49.2927614Z  -e PR_LABELS \ 2022-03-17T19:25:49.2928346Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-03-17T19:25:49.2929066Z  -e SCCACHE_BUCKET \ 2022-03-17T19:25:49.2929810Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-03-17T19:25:49.2930675Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-03-17T19:25:49.2931473Z  --ulimit stack=10485760:83886080 \ 2022-03-17T19:25:49.2932274Z  --security-opt seccomp=unconfined \ 2022-03-17T19:25:49.2933041Z  --cap-add=SYS_PTRACE \ 2022-03-17T19:25:49.2933759Z  --shm-size="${SHM_SIZE}" \ 2022-03-17T19:25:49.2934405Z  --tty \ 2022-03-17T19:25:49.2934989Z  --detach \ 2022-03-17T19:25:49.2935660Z  --name="${container_name}" \ 2022-03-17T19:25:49.2936343Z  --user jenkins \ 2022-03-17T19:25:49.2937167Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-03-17T19:25:49.2938032Z  -w /var/lib/jenkins/workspace \ 2022-03-17T19:25:49.2938735Z  "${DOCKER_IMAGE}" 2022-03-17T19:25:49.2939325Z ) 2022-03-17T19:25:49.2940339Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2022-03-17T19:25:49.2941841Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2022-03-17T19:25:49.2943246Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-03-17T19:25:49.2944562Z docker exec -t "${container_name}" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2022-03-17T19:25:49.2994440Z shell: /bin/bash -e {0} 2022-03-17T19:25:49.2994644Z env: 2022-03-17T19:25:49.2994922Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:25:49.2995446Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T19:25:49.2995868Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:25:49.2996252Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:25:49.2996569Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:25:49.2996786Z IN_CI: 1 2022-03-17T19:25:49.2996980Z IS_GHA: 1 2022-03-17T19:25:49.2997174Z IN_WHEEL_TEST: 1 2022-03-17T19:25:49.2997455Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:25:49.2997796Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:25:49.2998086Z PR_LABELS: [] 2022-03-17T19:25:49.2998439Z GITHUB_TOKEN: *** 2022-03-17T19:25:49.2998668Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:25:49.2998898Z PR_NUMBER: 2022-03-17T19:25:49.2999129Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:25:49.2999389Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:25:49.2999634Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:25:49.3000062Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:25:49.3000547Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T19:25:49.3000842Z TEST_CONFIG: distributed 2022-03-17T19:25:49.3001064Z SHARD_NUMBER: 1 2022-03-17T19:25:49.3001272Z NUM_TEST_SHARDS: 1 2022-03-17T19:25:49.3001485Z PR_BODY: 2022-03-17T19:25:49.3001801Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T19:25:49.3002277Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T19:25:49.3002604Z SHM_SIZE: 8g 2022-03-17T19:25:49.3002804Z BRANCH: master 2022-03-17T19:25:49.3003009Z ##[endgroup] 2022-03-17T19:25:49.3056909Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-03-17T19:25:49.3058525Z + [[ linux-bionic-rocm4.5-py3.7-distributed == *onnx* ]] 2022-03-17T19:25:49.3059425Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-03-17T19:25:49.3074145Z +++ nproc --ignore=2 2022-03-17T19:25:49.3095846Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e CUSTOM_TEST_ARTIFACT_BUILD_DIR -e GITHUB_ACTIONS -e IN_CI -e IS_GHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e JOB_BASE_NAME -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e PYTORCH_RETRY_TEST_CASES -e PR_LABELS -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2000264370 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:25:50.6385092Z + container_name=38dccec0eac565a8c82df885e30ce5584a016dd65e3050f93bf07338c6b019b2 2022-03-17T19:25:50.6387601Z + docker exec -t 38dccec0eac565a8c82df885e30ce5584a016dd65e3050f93bf07338c6b019b2 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-03-17T19:25:55.8611200Z Processing ./dist/torch-1.12.0a0+git4b4f652-cp37-cp37m-linux_x86_64.whl 2022-03-17T19:25:55.9638527Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.12.0a0+git4b4f652) (3.10.0.2) 2022-03-17T19:25:56.4248723Z Installing collected packages: torch 2022-03-17T19:26:04.8675834Z Successfully installed torch-1.12.0a0+git4b4f652 2022-03-17T19:26:04.9278254Z + COMPACT_JOB_NAME=linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T19:26:04.9280696Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-03-17T19:26:04.9517295Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch 2022-03-17T19:26:04.9519279Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T19:26:04.9520692Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib 2022-03-17T19:26:04.9522166Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test 2022-03-17T19:26:04.9522931Z + BUILD_DIR=build 2022-03-17T19:26:04.9523532Z + BUILD_RENAMED_DIR=build_renamed 2022-03-17T19:26:04.9524183Z + BUILD_BIN_DIR=build/bin 2022-03-17T19:26:04.9524932Z + [[ -n distributed ]] 2022-03-17T19:26:04.9526060Z + BUILD_ENVIRONMENT=linux-bionic-rocm4.5-py3.7-distributed-distributed 2022-03-17T19:26:04.9527469Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed != *bazel* ]] 2022-03-17T19:26:04.9528376Z ++ realpath build/custom_test_artifacts 2022-03-17T19:26:04.9540613Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2022-03-17T19:26:04.9544222Z ++ dirname .jenkins/pytorch/test.sh 2022-03-17T19:26:04.9558468Z + source .jenkins/pytorch/common.sh 2022-03-17T19:26:04.9563527Z +++ dirname .jenkins/pytorch/common.sh 2022-03-17T19:26:04.9592416Z ++ source .jenkins/pytorch/common_utils.sh 2022-03-17T19:26:04.9597780Z +++ TORCHVISION_COMMIT=8a2dc6f22ac4389ccba8859aa1e1cb14f1ee53db 2022-03-17T19:26:04.9599397Z ++ set -ex 2022-03-17T19:26:04.9615109Z ++++ dirname .jenkins/pytorch/common.sh 2022-03-17T19:26:04.9642837Z +++ cd .jenkins/pytorch 2022-03-17T19:26:04.9643668Z +++ pwd -P 2022-03-17T19:26:04.9644366Z ++ SCRIPT_DIR=/var/lib/jenkins/pytorch/.jenkins/pytorch 2022-03-17T19:26:04.9646213Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *rocm* ]] 2022-03-17T19:26:04.9648179Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed =~ py((2|3)\.?[0-9]?\.?[0-9]?) ]] 2022-03-17T19:26:04.9649129Z ++ unset HIP_PLATFORM 2022-03-17T19:26:04.9652653Z +++ which python3.7 2022-03-17T19:26:04.9683074Z ++ PYTHON=/opt/conda/bin/python3.7 2022-03-17T19:26:04.9684334Z ++ shopt -s expand_aliases 2022-03-17T19:26:04.9685243Z ++ export PYTORCH_TEST_WITH_ROCM=1 2022-03-17T19:26:04.9685960Z ++ PYTORCH_TEST_WITH_ROCM=1 2022-03-17T19:26:04.9686779Z ++ alias 'python=$PYTHON' 2022-03-17T19:26:04.9687411Z ++ export HSAKMT_DEBUG_LEVEL=4 2022-03-17T19:26:04.9688196Z ++ HSAKMT_DEBUG_LEVEL=4 2022-03-17T19:26:04.9688994Z ++ echo ENTERED_USER_LAND 2022-03-17T19:26:04.9689766Z ENTERED_USER_LAND 2022-03-17T19:26:04.9690492Z ++ export IN_CI=1 2022-03-17T19:26:04.9691019Z ++ IN_CI=1 2022-03-17T19:26:04.9691740Z ++ declare -f -t trap_add 2022-03-17T19:26:04.9692364Z ++ trap_add cleanup EXIT 2022-03-17T19:26:04.9692997Z ++ trap_add_cmd=cleanup 2022-03-17T19:26:04.9693560Z ++ shift 2022-03-17T19:26:04.9694091Z ++ for trap_add_name in "$@" 2022-03-17T19:26:04.9698948Z ++++ trap -p EXIT 2022-03-17T19:26:04.9703029Z +++ eval 'extract_trap_cmd ' 2022-03-17T19:26:04.9703831Z ++++ extract_trap_cmd 2022-03-17T19:26:04.9704782Z ++++ printf '%s\n' '' 2022-03-17T19:26:04.9705727Z +++ printf '%s\n' cleanup 2022-03-17T19:26:04.9709555Z ++ trap -- ' 2022-03-17T19:26:04.9710393Z cleanup' EXIT 2022-03-17T19:26:04.9715515Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed != *win-* ]] 2022-03-17T19:26:04.9716531Z ++ which sccache 2022-03-17T19:26:04.9735943Z ++ sccache --stop-server 2022-03-17T19:26:04.9769426Z ++ true 2022-03-17T19:26:04.9770642Z ++ rm -f /var/lib/jenkins/sccache_error.log 2022-03-17T19:26:04.9795558Z ++ [[ -n '' ]] 2022-03-17T19:26:04.9796572Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *rocm* ]] 2022-03-17T19:26:04.9797291Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2022-03-17T19:26:04.9797815Z ++ SCCACHE_IDLE_TIMEOUT=0 2022-03-17T19:26:04.9798346Z ++ sccache --start-server 2022-03-17T19:26:04.9822681Z Starting sccache server... 2022-03-17T19:26:05.6556395Z ++ sccache --zero-stats 2022-03-17T19:26:05.6588798Z Compile requests 0 2022-03-17T19:26:05.6589841Z Compile requests executed 0 2022-03-17T19:26:05.6590538Z Cache hits 0 2022-03-17T19:26:05.6591165Z Cache misses 0 2022-03-17T19:26:05.6591791Z Cache timeouts 0 2022-03-17T19:26:05.6592802Z Cache read errors 0 2022-03-17T19:26:05.6593460Z Forced recaches 0 2022-03-17T19:26:05.6594117Z Cache write errors 0 2022-03-17T19:26:05.6594852Z Compilation failures 0 2022-03-17T19:26:05.6595763Z Cache errors 0 2022-03-17T19:26:05.6596751Z Non-cacheable compilations 0 2022-03-17T19:26:05.6597609Z Non-cacheable calls 0 2022-03-17T19:26:05.6598434Z Non-compilation calls 0 2022-03-17T19:26:05.6599171Z Unsupported compiler calls 0 2022-03-17T19:26:05.6599865Z Average cache write 0.000 s 2022-03-17T19:26:05.6600566Z Average cache read miss 0.000 s 2022-03-17T19:26:05.6601254Z Average cache read hit 0.000 s 2022-03-17T19:26:05.6603588Z Cache location S3, bucket: Bucket(name=ossci-compiler-cache-circleci-v2, base_url=http://ossci-compiler-cache-circleci-v2.s3.amazonaws.com/) 2022-03-17T19:26:05.6605342Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-test == *-build ]] 2022-03-17T19:26:05.6606139Z ++ which ccache 2022-03-17T19:26:05.6615404Z ++ '[' -z linux-bionic-rocm4.5-py3.7-distributed ']' 2022-03-17T19:26:05.6616888Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *linux-trusty-py3.6-gcc7* ]] 2022-03-17T19:26:05.6617949Z ++ BUILD_TEST_LIBTORCH=0 2022-03-17T19:26:05.6619153Z ++ [[ distributed == *xla* ]] 2022-03-17T19:26:05.6620349Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *centos* ]] 2022-03-17T19:26:05.6622148Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *linux-bionic* ]] 2022-03-17T19:26:05.6623072Z ++ which conda 2022-03-17T19:26:05.6640495Z /opt/conda/bin/conda 2022-03-17T19:26:05.6642012Z ++ conda install -q -y cmake 2022-03-17T19:26:07.9941809Z Collecting package metadata (current_repodata.json): ...working... done 2022-03-17T19:26:08.4828257Z Solving environment: ...working... done 2022-03-17T19:26:08.5810459Z 2022-03-17T19:26:08.5811215Z ## Package Plan ## 2022-03-17T19:26:08.5811736Z 2022-03-17T19:26:08.5812184Z environment location: /opt/conda 2022-03-17T19:26:08.5812847Z 2022-03-17T19:26:08.5813241Z added / updated specs: 2022-03-17T19:26:08.5814179Z - cmake 2022-03-17T19:26:08.5814509Z 2022-03-17T19:26:08.5814540Z 2022-03-17T19:26:08.5814893Z The following packages will be downloaded: 2022-03-17T19:26:08.5815367Z 2022-03-17T19:26:08.5815661Z package | build 2022-03-17T19:26:08.5816594Z ---------------------------|----------------- 2022-03-17T19:26:08.5817612Z bzip2-1.0.8 | h7b6447c_0 78 KB 2022-03-17T19:26:08.5818606Z c-ares-1.18.1 | h7f8727e_0 114 KB 2022-03-17T19:26:08.5819693Z cmake-3.22.1 | h1fce559_0 7.3 MB 2022-03-17T19:26:08.5821202Z expat-2.4.4 | h295c915_0 169 KB 2022-03-17T19:26:08.5822454Z krb5-1.19.2 | hac12032_0 1.2 MB 2022-03-17T19:26:08.5823726Z libcurl-7.80.0 | h0b77cf5_0 339 KB 2022-03-17T19:26:08.5824804Z libedit-3.1.20210910 | h7f8727e_0 166 KB 2022-03-17T19:26:08.5825775Z libev-4.33 | h7f8727e_1 111 KB 2022-03-17T19:26:08.5826780Z libnghttp2-1.46.0 | hce63b2e_0 680 KB 2022-03-17T19:26:08.5827768Z libssh2-1.9.0 | h1ba5d50_1 269 KB 2022-03-17T19:26:08.5828757Z libuv-1.40.0 | h7b6447c_0 736 KB 2022-03-17T19:26:08.5829735Z lz4-c-1.9.3 | h295c915_1 185 KB 2022-03-17T19:26:08.5830677Z rhash-1.4.1 | h3c74f83_1 203 KB 2022-03-17T19:26:08.5831687Z zstd-1.5.0 | ha4553b6_1 524 KB 2022-03-17T19:26:08.5832709Z ------------------------------------------------------------ 2022-03-17T19:26:08.5833470Z Total: 12.0 MB 2022-03-17T19:26:08.5833869Z 2022-03-17T19:26:08.5834677Z The following NEW packages will be INSTALLED: 2022-03-17T19:26:08.5835181Z 2022-03-17T19:26:08.5867124Z bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 2022-03-17T19:26:08.5868500Z c-ares pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 2022-03-17T19:26:08.5869763Z cmake pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 2022-03-17T19:26:08.5870983Z expat pkgs/main/linux-64::expat-2.4.4-h295c915_0 2022-03-17T19:26:08.5872194Z krb5 pkgs/main/linux-64::krb5-1.19.2-hac12032_0 2022-03-17T19:26:08.5873450Z libcurl pkgs/main/linux-64::libcurl-7.80.0-h0b77cf5_0 2022-03-17T19:26:08.5874747Z libedit pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 2022-03-17T19:26:08.5875997Z libev pkgs/main/linux-64::libev-4.33-h7f8727e_1 2022-03-17T19:26:08.5877264Z libnghttp2 pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 2022-03-17T19:26:08.5878570Z libssh2 pkgs/main/linux-64::libssh2-1.9.0-h1ba5d50_1 2022-03-17T19:26:08.5879784Z libuv pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 2022-03-17T19:26:08.5880946Z lz4-c pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 2022-03-17T19:26:08.5882293Z rhash pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 2022-03-17T19:26:08.5883495Z zstd pkgs/main/linux-64::zstd-1.5.0-ha4553b6_1 2022-03-17T19:26:08.5883985Z 2022-03-17T19:26:08.5884001Z 2022-03-17T19:26:10.5238325Z Preparing transaction: ...working... done 2022-03-17T19:26:10.9615889Z Verifying transaction: ...working... done 2022-03-17T19:26:11.5085316Z Executing transaction: ...working... done 2022-03-17T19:26:11.6940974Z ++ [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *centos* ]] 2022-03-17T19:26:11.6942435Z + echo 'Testing pytorch' 2022-03-17T19:26:11.6943255Z Testing pytorch 2022-03-17T19:26:11.6943949Z + export LANG=C.UTF-8 2022-03-17T19:26:11.6944594Z + LANG=C.UTF-8 2022-03-17T19:26:11.6945157Z + PR_NUMBER= 2022-03-17T19:26:11.6945779Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-03-17T19:26:11.6946971Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-slow-* ]] 2022-03-17T19:26:11.6947900Z + [[ distributed == \s\l\o\w ]] 2022-03-17T19:26:11.6949139Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *slow-gradcheck* ]] 2022-03-17T19:26:11.6950597Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *cuda* ]] 2022-03-17T19:26:11.6952013Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *cuda11* ]] 2022-03-17T19:26:11.6953445Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *noarch* ]] 2022-03-17T19:26:11.6954392Z + export PYTORCH_TEST_SKIP_NOARCH=1 2022-03-17T19:26:11.6955095Z + PYTORCH_TEST_SKIP_NOARCH=1 2022-03-17T19:26:11.6955799Z + [[ -n '' ]] 2022-03-17T19:26:11.6956474Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0 2022-03-17T19:26:11.6957284Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0 2022-03-17T19:26:11.6958483Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *rocm* ]] 2022-03-17T19:26:11.6959300Z + rocminfo 2022-03-17T19:26:11.6960088Z + grep -E 'Name:.*\sgfx|Marketing' 2022-03-17T19:26:11.7791252Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:26:11.7792785Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:26:11.7793998Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:26:11.7795175Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-03-17T19:26:11.7796055Z Name: gfx906 2022-03-17T19:26:11.7796811Z Marketing Name: 2022-03-17T19:26:11.7797583Z Name: gfx906 2022-03-17T19:26:11.7798348Z Marketing Name: 2022-03-17T19:26:11.7986012Z + [[ -z true ]] 2022-03-17T19:26:11.7987200Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed != *ppc64le* ]] 2022-03-17T19:26:11.7988931Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed != *-bazel-* ]] 2022-03-17T19:26:11.7989997Z + pip_install --user ninja 2022-03-17T19:26:11.7990925Z + pip install --progress-bar off --user ninja 2022-03-17T19:26:12.3548412Z Collecting ninja 2022-03-17T19:26:12.4478865Z Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-03-17T19:26:12.4720975Z [?25l 2022-03-17T19:26:12.8641615Z [?25hInstalling collected packages: ninja 2022-03-17T19:26:12.8741852Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-03-17T19:26:12.8743632Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T19:26:12.8771400Z Successfully installed ninja-1.10.2.3 2022-03-17T19:26:12.9422543Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-03-17T19:26:12.9424448Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-03-17T19:26:12.9426382Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *asan* ]] 2022-03-17T19:26:12.9427813Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-NO_AVX-* ]] 2022-03-17T19:26:12.9429165Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X ]] 2022-03-17T19:26:12.9430445Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-NO_AVX2-* ]] 2022-03-17T19:26:12.9431445Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-03-17T19:26:12.9432704Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-NO_AVX512-* ]] 2022-03-17T19:26:12.9433698Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\5\1\2 ]] 2022-03-17T19:26:12.9434928Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *tbb* ]] 2022-03-17T19:26:12.9441162Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *libtorch* ]] 2022-03-17T19:26:12.9442777Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-bazel-* ]] 2022-03-17T19:26:12.9444211Z + cd test 2022-03-17T19:26:12.9445680Z + /opt/conda/bin/python3.7 -c 'import torch; print(torch.__config__.show())' 2022-03-17T19:26:13.7133403Z PyTorch built with: 2022-03-17T19:26:13.7133764Z - GCC 9.4 2022-03-17T19:26:13.7134170Z - C++ Version: 201402 2022-03-17T19:26:13.7134945Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-03-17T19:26:13.7135591Z - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e) 2022-03-17T19:26:13.7136043Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-03-17T19:26:13.7136468Z - LAPACK is enabled (usually provided by MKL) 2022-03-17T19:26:13.7136823Z - NNPACK is enabled 2022-03-17T19:26:13.7137170Z - CPU capability usage: AVX2 2022-03-17T19:26:13.7137514Z - HIP Runtime 4.4.21432 2022-03-17T19:26:13.7137834Z - MIOpen 2.14.0 2022-03-17T19:26:13.7138130Z - Magma 2.6.1 2022-03-17T19:26:13.7142041Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, 2022-03-17T19:26:13.7144684Z 2022-03-17T19:26:15.4951010Z + cd test 2022-03-17T19:26:15.4952350Z + /opt/conda/bin/python3.7 -c 'import torch; print(torch.__config__.parallel_info())' 2022-03-17T19:26:16.2390845Z ATen/Parallel: 2022-03-17T19:26:16.2391113Z at::get_num_threads() : 32 2022-03-17T19:26:16.2391430Z at::get_num_interop_threads() : 32 2022-03-17T19:26:16.2391779Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-03-17T19:26:16.2392185Z omp_get_max_threads() : 32 2022-03-17T19:26:16.2392884Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-03-17T19:26:16.2393317Z mkl_get_max_threads() : 32 2022-03-17T19:26:16.2393794Z Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e) 2022-03-17T19:26:16.2394185Z std::thread::hardware_concurrency() : 64 2022-03-17T19:26:16.2394492Z Environment variables: 2022-03-17T19:26:16.2394767Z OMP_NUM_THREADS : [not set] 2022-03-17T19:26:16.2395052Z MKL_NUM_THREADS : [not set] 2022-03-17T19:26:16.2395343Z ATen parallel backend: OpenMP 2022-03-17T19:26:16.2395531Z 2022-03-17T19:26:17.8691933Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *deploy* ]] 2022-03-17T19:26:17.8694484Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *backward* ]] 2022-03-17T19:26:17.8695563Z + [[ distributed == *xla* ]] 2022-03-17T19:26:17.8696814Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *jit_legacy-test ]] 2022-03-17T19:26:17.8698274Z + [[ linux-bionic-rocm4.5-py3.7-distributed-test == *jit_legacy-test ]] 2022-03-17T19:26:17.8699216Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-03-17T19:26:17.8700430Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *libtorch* ]] 2022-03-17T19:26:17.8701855Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-test1 ]] 2022-03-17T19:26:17.8703162Z + [[ linux-bionic-rocm4.5-py3.7-distributed-test == *-test1 ]] 2022-03-17T19:26:17.8703947Z + [[ 1 == 1 ]] 2022-03-17T19:26:17.8704612Z + [[ 1 -gt 1 ]] 2022-03-17T19:26:17.8705695Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-test2 ]] 2022-03-17T19:26:17.8707023Z + [[ linux-bionic-rocm4.5-py3.7-distributed-test == *-test2 ]] 2022-03-17T19:26:17.8707808Z + [[ 1 == 2 ]] 2022-03-17T19:26:17.8708450Z + [[ 1 -gt 2 ]] 2022-03-17T19:26:17.8709542Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *vulkan* ]] 2022-03-17T19:26:17.8710969Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *-bazel-* ]] 2022-03-17T19:26:17.8712434Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *distributed* ]] 2022-03-17T19:26:17.8713323Z + test_distributed 2022-03-17T19:26:17.8714212Z + echo 'Testing distributed python tests' 2022-03-17T19:26:17.8714970Z Testing distributed python tests 2022-03-17T19:26:17.8716134Z + /opt/conda/bin/python3.7 test/run_test.py --distributed-tests --verbose 2022-03-17T19:26:21.3110970Z Ignoring disabled issues: [] 2022-03-17T19:26:21.3245799Z Excluding distributed/nn/jit/test_instantiator on ROCm 2022-03-17T19:26:21.3247649Z Excluding distributed/rpc/test_faulty_agent on ROCm 2022-03-17T19:26:21.3248831Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2022-03-17T19:26:21.3250175Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2022-03-17T19:26:21.3251437Z Excluding distributed/_shard/sharded_tensor/test_megatron_prototype on ROCm 2022-03-17T19:26:21.3252788Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor on ROCm 2022-03-17T19:26:21.3254057Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor_reshard on ROCm 2022-03-17T19:26:21.3255757Z Excluding distributed/_shard/sharded_tensor/test_partial_tensor on ROCm 2022-03-17T19:26:21.3256803Z Excluding distributed/_shard/sharded_tensor/ops/test_elementwise_ops on ROCm 2022-03-17T19:26:21.3257800Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm 2022-03-17T19:26:21.3259060Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm 2022-03-17T19:26:21.3260044Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm 2022-03-17T19:26:21.3260995Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm 2022-03-17T19:26:21.3261926Z Excluding distributed/_shard/sharded_tensor/ops/test_linear on ROCm 2022-03-17T19:26:21.3262874Z Excluding distributed/_shard/sharding_spec/test_sharding_spec on ROCm 2022-03-17T19:26:21.3263808Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm 2022-03-17T19:26:21.3264745Z Excluding distributed/fsdp/test_flatten_params_wrapper on ROCm 2022-03-17T19:26:21.3265611Z Excluding distributed/fsdp/test_fsdp_apply on ROCm 2022-03-17T19:26:21.3266457Z Excluding distributed/fsdp/test_fsdp_checkpoint on ROCm 2022-03-17T19:26:21.3267339Z Excluding distributed/fsdp/test_fsdp_clip_grad_norm on ROCm 2022-03-17T19:26:21.3268165Z Excluding distributed/fsdp/test_fsdp_comm on ROCm 2022-03-17T19:26:21.3268981Z Excluding distributed/fsdp/test_fsdp_core on ROCm 2022-03-17T19:26:21.3269846Z Excluding distributed/fsdp/test_fsdp_freezing_weights on ROCm 2022-03-17T19:26:21.3270697Z Excluding distributed/fsdp/test_fsdp_grad_acc on ROCm 2022-03-17T19:26:21.3271522Z Excluding distributed/fsdp/test_fsdp_input on ROCm 2022-03-17T19:26:21.3272569Z Excluding distributed/fsdp/test_fsdp_memory on ROCm 2022-03-17T19:26:21.3273437Z Excluding distributed/fsdp/test_fsdp_multiple_forward on ROCm 2022-03-17T19:26:21.3274341Z Excluding distributed/fsdp/test_fsdp_multiple_wrapping on ROCm 2022-03-17T19:26:21.3275222Z Excluding distributed/fsdp/test_fsdp_overlap on ROCm 2022-03-17T19:26:21.3276062Z Excluding distributed/fsdp/test_fsdp_pure_fp16 on ROCm 2022-03-17T19:26:21.3276899Z Excluding distributed/fsdp/test_fsdp_state_dict on ROCm 2022-03-17T19:26:21.3277782Z Excluding distributed/fsdp/test_fsdp_summon_full_params on ROCm 2022-03-17T19:26:21.3278647Z Excluding distributed/fsdp/test_fsdp_traversal on ROCm 2022-03-17T19:26:21.3279478Z Excluding distributed/fsdp/test_fsdp_uneven on ROCm 2022-03-17T19:26:21.3280279Z Excluding distributed/fsdp/test_utils on ROCm 2022-03-17T19:26:21.3281035Z Excluding distributed/fsdp/test_wrap on ROCm 2022-03-17T19:26:21.3281689Z Selected tests: 2022-03-17T19:26:21.3282769Z distributed/algorithms/test_join 2022-03-17T19:26:21.3283459Z distributed/elastic/events/lib_test 2022-03-17T19:26:21.3284150Z distributed/elastic/metrics/api_test 2022-03-17T19:26:21.3284920Z distributed/elastic/multiprocessing/api_test 2022-03-17T19:26:21.3285660Z distributed/elastic/timer/api_test 2022-03-17T19:26:21.3286385Z distributed/elastic/timer/local_timer_example 2022-03-17T19:26:21.3287148Z distributed/elastic/timer/local_timer_test 2022-03-17T19:26:21.3287894Z distributed/elastic/utils/distributed_test 2022-03-17T19:26:21.3288643Z distributed/elastic/utils/logging_test 2022-03-17T19:26:21.3289343Z distributed/elastic/utils/util_test 2022-03-17T19:26:21.3290112Z distributed/optim/test_zero_redundancy_optimizer 2022-03-17T19:26:21.3290897Z distributed/pipeline/sync/skip/test_api 2022-03-17T19:26:21.3291643Z distributed/pipeline/sync/skip/test_gpipe 2022-03-17T19:26:21.3292471Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-03-17T19:26:21.3293262Z distributed/pipeline/sync/skip/test_leak 2022-03-17T19:26:21.3294032Z distributed/pipeline/sync/skip/test_portal 2022-03-17T19:26:21.3294817Z distributed/pipeline/sync/skip/test_stash_pop 2022-03-17T19:26:21.3295586Z distributed/pipeline/sync/skip/test_tracker 2022-03-17T19:26:21.3296415Z distributed/pipeline/sync/skip/test_verify_skippables 2022-03-17T19:26:21.3297193Z distributed/pipeline/sync/test_balance 2022-03-17T19:26:21.3297937Z distributed/pipeline/sync/test_bugs 2022-03-17T19:26:21.3298845Z distributed/pipeline/sync/test_checkpoint 2022-03-17T19:26:21.3299598Z distributed/pipeline/sync/test_copy 2022-03-17T19:26:21.3300378Z distributed/pipeline/sync/test_deferred_batch_norm 2022-03-17T19:26:21.3301163Z distributed/pipeline/sync/test_dependency 2022-03-17T19:26:21.3301904Z distributed/pipeline/sync/test_inplace 2022-03-17T19:26:21.3302642Z distributed/pipeline/sync/test_microbatch 2022-03-17T19:26:21.3303384Z distributed/pipeline/sync/test_phony 2022-03-17T19:26:21.3304101Z distributed/pipeline/sync/test_pipe 2022-03-17T19:26:21.3304846Z distributed/pipeline/sync/test_pipeline 2022-03-17T19:26:21.3305589Z distributed/pipeline/sync/test_stream 2022-03-17T19:26:21.3306333Z distributed/pipeline/sync/test_transparency 2022-03-17T19:26:21.3307085Z distributed/pipeline/sync/test_worker 2022-03-17T19:26:21.3307749Z distributed/test_c10d_common 2022-03-17T19:26:21.3308378Z distributed/test_c10d_gloo 2022-03-17T19:26:21.3308987Z distributed/test_c10d_nccl 2022-03-17T19:26:21.3309650Z distributed/test_c10d_spawn_gloo 2022-03-17T19:26:21.3310318Z distributed/test_c10d_spawn_nccl 2022-03-17T19:26:21.3310964Z distributed/test_data_parallel 2022-03-17T19:26:21.3311644Z distributed/test_distributed_spawn 2022-03-17T19:26:21.3312293Z distributed/test_launcher 2022-03-17T19:26:21.3312910Z distributed/test_nccl 2022-03-17T19:26:21.3313512Z distributed/test_pg_wrapper 2022-03-17T19:26:21.3314134Z distributed/test_store 2022-03-17T19:26:21.3328888Z Prioritized test from test file changes. 2022-03-17T19:26:21.3330169Z reordering tests for PR: 2022-03-17T19:26:21.3330938Z prioritized: [] 2022-03-17T19:26:21.3342234Z the rest: ['distributed/algorithms/test_join', 'distributed/elastic/events/lib_test', 'distributed/elastic/metrics/api_test', 'distributed/elastic/multiprocessing/api_test', 'distributed/elastic/timer/api_test', 'distributed/elastic/timer/local_timer_example', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/utils/distributed_test', 'distributed/elastic/utils/logging_test', 'distributed/elastic/utils/util_test', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_worker', 'distributed/test_c10d_common', 'distributed/test_c10d_gloo', 'distributed/test_c10d_nccl', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_spawn_nccl', 'distributed/test_data_parallel', 'distributed/test_distributed_spawn', 'distributed/test_launcher', 'distributed/test_nccl', 'distributed/test_pg_wrapper', 'distributed/test_store'] 2022-03-17T19:26:21.3349044Z 2022-03-17T19:26:21.8956447Z Running distributed/algorithms/test_join ... [2022-03-17 19:26:21.894816] 2022-03-17T19:26:21.8958590Z Executing ['/opt/conda/bin/python3.7', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:26:21.894906] 2022-03-17T19:26:22.8064648Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-03-17T19:26:22.8080101Z 2022-03-17T19:26:22.8080751Z Running tests... 2022-03-17T19:26:22.8082579Z ---------------------------------------------------------------------- 2022-03-17T19:26:22.8093986Z test_join_kwargs (__main__.TestJoin) 2022-03-17T19:26:23.4422895Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 809 2022-03-17T19:26:23.4453867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 810 2022-03-17T19:26:24.3385163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:24.3385797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:24.3701963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:24.3703358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:24.3704369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:24.3799380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:24.7448507Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:24.7989833Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:26.4549310Z ok (3.646s) 2022-03-17T19:26:26.4571837Z test_multiple_joinable_disable (__main__.TestJoin) 2022-03-17T19:26:26.4626664Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 954 2022-03-17T19:26:26.4662223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 955 2022-03-17T19:26:27.3575843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:27.3576313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:27.3779894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:27.3781975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:27.3782534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:27.3783299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:27.7564035Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:27.8021045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:30.2735437Z ok (3.818s) 2022-03-17T19:26:30.2759806Z test_multiple_joinables (__main__.TestJoin) 2022-03-17T19:26:30.2821648Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1099 2022-03-17T19:26:30.2853362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1100 2022-03-17T19:26:31.1208720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:31.1209951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:31.1371310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:31.1374282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:31.1376357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:31.1421169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:31.5184459Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:31.5804592Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:33.8945780Z ok (3.621s) 2022-03-17T19:26:33.8961101Z test_multiple_joinables_throw (__main__.TestJoin) 2022-03-17T19:26:33.9011122Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1244 2022-03-17T19:26:33.9059324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1245 2022-03-17T19:26:34.8223774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:34.8224240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:34.8322167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:34.8323248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:34.8324666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:34.8326057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:35.2197330Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:35.2664435Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:37.5144362Z ok (3.620s) 2022-03-17T19:26:37.5169062Z test_single_joinable (__main__.TestJoin) 2022-03-17T19:26:37.5225556Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1389 2022-03-17T19:26:37.5270161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1390 2022-03-17T19:26:38.4619956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:38.4623137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:38.4677405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:38.4679827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:38.4681313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:38.4733955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:38.8593652Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:38.9149290Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:40.9354406Z ok (3.421s) 2022-03-17T19:26:40.9374033Z test_single_joinable_disable (__main__.TestJoin) 2022-03-17T19:26:40.9429261Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1534 2022-03-17T19:26:40.9484485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1535 2022-03-17T19:26:41.8782456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:41.8782918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:41.8796404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:41.8798543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:41.8799397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:41.8887682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:42.2813647Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:42.3313100Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:43.9546966Z ok (3.019s) 2022-03-17T19:26:43.9567517Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-03-17T19:26:43.9634235Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1679 2022-03-17T19:26:43.9675362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1680 2022-03-17T19:26:44.8232302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:44.8232754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:44.8761145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:44.8761795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:44.8762695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:44.8837872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:45.2569775Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:45.2983028Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:47.2790759Z ok (3.324s) 2022-03-17T19:26:47.2808845Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-03-17T19:26:47.2875694Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1824 2022-03-17T19:26:47.2915115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1825 2022-03-17T19:26:48.1966422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:48.1966907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:48.2115850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:48.2118156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:48.2118994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:48.2171843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:48.5993171Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:48.6549799Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:50.5004496Z ok (3.221s) 2022-03-17T19:26:50.5021549Z test_single_joinable_throw (__main__.TestJoin) 2022-03-17T19:26:50.5085504Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1969 2022-03-17T19:26:50.5126866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1970 2022-03-17T19:26:51.4054750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:51.4057391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:51.4512432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:51.4514259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:51.4515899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:51.4568394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:51.8386279Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:51.8970871Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:26:54.0217424Z ok (3.521s) 2022-03-17T19:26:54.0217814Z 2022-03-17T19:26:54.0218774Z ---------------------------------------------------------------------- 2022-03-17T19:26:54.0219555Z Ran 9 tests in 31.214s 2022-03-17T19:26:54.0223599Z 2022-03-17T19:26:54.0224644Z OK 2022-03-17T19:26:54.0225127Z 2022-03-17T19:26:54.0225428Z Generating XML reports... 2022-03-17T19:26:54.0315808Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220317192622.xml 2022-03-17T19:26:55.6128481Z Running distributed/elastic/events/lib_test ... [2022-03-17 19:26:55.611957] 2022-03-17T19:26:55.6130150Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-03-17 19:26:55.612051] 2022-03-17T19:26:56.1541308Z ============================= test session starts ============================== 2022-03-17T19:26:56.1542629Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:26:56.1600513Z cachedir: .pytest_cache 2022-03-17T19:26:56.1602491Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:26:56.1604186Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:26:56.1605136Z plugins: hypothesis-4.53.2 2022-03-17T19:26:56.9238535Z collecting ...  2022-03-17T19:26:56.9250038Z collecting 3 items  2022-03-17T19:26:56.9250453Z collected 8 items  2022-03-17T19:26:56.9252757Z 2022-03-17T19:26:56.9266431Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-03-17T19:26:56.9277433Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-03-17T19:26:56.9291419Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-03-17T19:26:56.9867335Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-03-17T19:26:56.9881081Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-03-17T19:26:56.9890776Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-03-17T19:26:56.9901799Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-03-17T19:26:56.9920303Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-03-17T19:26:56.9920596Z 2022-03-17T19:26:56.9920865Z ============================== 8 passed in 0.84s =============================== 2022-03-17T19:26:58.9046313Z Running distributed/elastic/metrics/api_test ... [2022-03-17 19:26:58.903804] 2022-03-17T19:26:58.9048238Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:26:58.903929] 2022-03-17T19:26:59.7371133Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-03-17T19:26:59.7383870Z 2022-03-17T19:26:59.7384444Z Running tests... 2022-03-17T19:26:59.7385646Z ---------------------------------------------------------------------- 2022-03-17T19:27:00.3196672Z test_get_metric_name (__main__.MetricsApiTest) ... ok (0.581s) 2022-03-17T19:27:00.3209267Z test_inheritance (__main__.MetricsApiTest) ... ok (0.001s) 2022-03-17T19:27:00.3225949Z test_profile (__main__.MetricsApiTest) ... ok (0.002s) 2022-03-17T19:27:00.3226485Z 2022-03-17T19:27:00.3227296Z ---------------------------------------------------------------------- 2022-03-17T19:27:00.3228084Z Ran 3 tests in 0.584s 2022-03-17T19:27:00.3228466Z 2022-03-17T19:27:00.3228649Z OK 2022-03-17T19:27:00.3228944Z 2022-03-17T19:27:00.3229221Z Generating XML reports... 2022-03-17T19:27:00.3261155Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220317192659.xml 2022-03-17T19:27:01.8281601Z Running distributed/elastic/multiprocessing/api_test ... [2022-03-17 19:27:01.827004] 2022-03-17T19:27:01.8283983Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:01.827165] 2022-03-17T19:27:02.8159965Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-03-17T19:27:02.8179419Z 2022-03-17T19:27:02.8179842Z Running tests... 2022-03-17T19:27:02.8180407Z ---------------------------------------------------------------------- 2022-03-17T19:27:03.4012640Z test_get_failures (__main__.RunProcResultsTest) ... ok (0.583s) 2022-03-17T19:27:03.4020189Z test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s) 2022-03-17T19:27:03.4038771Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:27:03.4426869Z test_binary (__main__.StartProcessesListTest) ... hello stdout from 0 2022-03-17T19:27:03.4427993Z hello stderr from 0 2022-03-17T19:27:03.4471021Z hello stdout from 1 2022-03-17T19:27:03.4471659Z hello stderr from 1 2022-03-17T19:27:03.5249335Z ok (0.121s) 2022-03-17T19:27:03.5780905Z test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1 2022-03-17T19:27:03.5781747Z bar stderr from 1 2022-03-17T19:27:03.6532301Z failed (exitcode: 138) local_rank: 0 (pid: 2317) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-03-17T19:27:03.6555064Z ok (0.130s) 2022-03-17T19:27:03.6712096Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.015s) 2022-03-17T19:27:03.7175259Z test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last): 2022-03-17T19:27:03.7176619Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-03-17T19:27:03.7177554Z raise RuntimeError(f"raised from {rank}") 2022-03-17T19:27:03.7178299Z RuntimeError: raised from 0 2022-03-17T19:27:03.7291377Z bar from 1 2022-03-17T19:27:03.8031902Z failed (exitcode: 1) local_rank: 0 (pid: 2320) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-03-17T19:27:03.8054176Z ok (0.134s) 2022-03-17T19:27:03.8607353Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1 2022-03-17T19:27:03.9344956Z [trainer0]:hello stdout from 0 2022-03-17T19:27:03.9345793Z [trainer1]:world stderr from 1 2022-03-17T19:27:04.9383410Z ok (1.132s) 2022-03-17T19:27:05.8991286Z test_function (__main__.StartProcessesListTest) ... hello stdout from 1 2022-03-17T19:27:05.8991682Z hello stderr from 1 2022-03-17T19:27:05.9706651Z hello stdout from 0 2022-03-17T19:27:05.9706924Z hello stderr from 0 2022-03-17T19:27:07.5356136Z Closing process 2327 via signal SIGTERM 2022-03-17T19:27:07.5706908Z ok (2.632s) 2022-03-17T19:27:10.5953260Z test_function_large_ret_val (__main__.StartProcessesListTest) ... Closing process 2461 via signal SIGTERM 2022-03-17T19:27:10.5956193Z Closing process 2463 via signal SIGTERM 2022-03-17T19:27:10.5962877Z Closing process 2464 via signal SIGTERM 2022-03-17T19:27:10.6598309Z ok (3.089s) 2022-03-17T19:27:10.6625663Z test_function_raise (__main__.StartProcessesListTest) 2022-03-17T19:27:13.2164761Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 2729) of fn: echo2 (start_method: spawn) 2022-03-17T19:27:13.2166541Z Traceback (most recent call last): 2022-03-17T19:27:13.2168450Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:27:13.2169563Z self._pc.join(-1) 2022-03-17T19:27:13.2170789Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-03-17T19:27:13.2172172Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-03-17T19:27:13.2173408Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-03-17T19:27:13.2174053Z 2022-03-17T19:27:13.2174658Z -- Process 0 terminated with the following error: 2022-03-17T19:27:13.2175435Z Traceback (most recent call last): 2022-03-17T19:27:13.2176722Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-03-17T19:27:13.2177595Z fn(i, *args) 2022-03-17T19:27:13.2178911Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap 2022-03-17T19:27:13.2179884Z ret = record(fn)(*args_) 2022-03-17T19:27:13.2181339Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper 2022-03-17T19:27:13.2182337Z return f(*args, **kwargs) 2022-03-17T19:27:13.2183295Z File "/var/lib/jenkins/pytorch/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-03-17T19:27:13.2184223Z raise RuntimeError(msg) 2022-03-17T19:27:13.2184861Z RuntimeError: hello 2022-03-17T19:27:13.2185551Z 2022-03-17T19:27:13.2190332Z ok (2.559s) 2022-03-17T19:27:13.2247878Z test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.005s) 2022-03-17T19:27:13.2266915Z test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:27:13.2337266Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 2863 via signal SIGTERM 2022-03-17T19:27:13.2403970Z ok (0.013s) 2022-03-17T19:27:13.2458559Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-03-17T19:27:13.2460049Z Traceback (most recent call last): 2022-03-17T19:27:13.2461540Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:27:13.2462628Z self._pc.join(-1) 2022-03-17T19:27:13.2463390Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-03-17T19:27:13.2464264Z return _mock_self._mock_call(*args, **kwargs) 2022-03-17T19:27:13.2465334Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-03-17T19:27:13.2466225Z raise effect 2022-03-17T19:27:13.2467269Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-03-17T19:27:13.2468242Z ok (0.006s) 2022-03-17T19:27:16.7164593Z test_pcontext_wait (__main__.StartProcessesListTest) ... ok (3.469s) 2022-03-17T19:27:16.7335613Z test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 2931 closing signal SIGTERM 2022-03-17T19:27:16.7358022Z ok (0.019s) 2022-03-17T19:27:16.7410828Z test_to_map (__main__.StartProcessesListTest) ... ok (0.005s) 2022-03-17T19:27:16.7429986Z test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:27:17.7826323Z test_void_function (__main__.StartProcessesListTest) ... world 2022-03-17T19:27:17.7984188Z hello 2022-03-17T19:27:19.5149123Z Closing process 2933 via signal SIGTERM 2022-03-17T19:27:19.5339057Z ok (2.791s) 2022-03-17T19:27:19.5393281Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.005s) 2022-03-17T19:27:19.5896682Z test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1 2022-03-17T19:27:19.5897555Z bar stderr from 1 2022-03-17T19:27:19.6656794Z failed (exitcode: 138) local_rank: 0 (pid: 3066) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-03-17T19:27:19.6677510Z ok (0.128s) 2022-03-17T19:27:19.6830616Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.015s) 2022-03-17T19:27:19.7236294Z test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last): 2022-03-17T19:27:19.7237685Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-03-17T19:27:19.7238796Z raise RuntimeError(f"raised from {rank}") 2022-03-17T19:27:19.7239635Z RuntimeError: raised from 0 2022-03-17T19:27:19.7344188Z bar from 1 2022-03-17T19:27:19.8091731Z failed (exitcode: 1) local_rank: 0 (pid: 3069) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-03-17T19:27:19.8111490Z ok (0.128s) 2022-03-17T19:27:22.8322570Z test_function_large_ret_val (__main__.StartProcessesTest) ... Closing process 3071 via signal SIGTERM 2022-03-17T19:27:22.8323688Z Closing process 3072 via signal SIGTERM 2022-03-17T19:27:22.8334026Z Closing process 3074 via signal SIGTERM 2022-03-17T19:27:22.8986328Z ok (3.087s) 2022-03-17T19:27:22.9008629Z test_function_raise (__main__.StartProcessesTest) 2022-03-17T19:27:25.5618352Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 3339) of fn: echo2 (start_method: spawn) 2022-03-17T19:27:25.5619824Z Traceback (most recent call last): 2022-03-17T19:27:25.5621554Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:27:25.5622657Z self._pc.join(-1) 2022-03-17T19:27:25.5624506Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-03-17T19:27:25.5625640Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-03-17T19:27:25.5626693Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-03-17T19:27:25.5627321Z 2022-03-17T19:27:25.5627914Z -- Process 0 terminated with the following error: 2022-03-17T19:27:25.5628694Z Traceback (most recent call last): 2022-03-17T19:27:25.5629955Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-03-17T19:27:25.5630810Z fn(i, *args) 2022-03-17T19:27:25.5632090Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap 2022-03-17T19:27:25.5633058Z ret = record(fn)(*args_) 2022-03-17T19:27:25.5634470Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper 2022-03-17T19:27:25.5635473Z return f(*args, **kwargs) 2022-03-17T19:27:25.5636427Z File "/var/lib/jenkins/pytorch/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-03-17T19:27:25.5637341Z raise RuntimeError(msg) 2022-03-17T19:27:25.5637972Z RuntimeError: hello 2022-03-17T19:27:25.5638347Z 2022-03-17T19:27:25.5646541Z ok (2.666s) 2022-03-17T19:27:25.5704907Z test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.005s) 2022-03-17T19:27:25.5743710Z test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.004s) 2022-03-17T19:27:25.5809738Z test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 3473 via signal SIGTERM 2022-03-17T19:27:25.5876335Z ok (0.013s) 2022-03-17T19:27:25.5952971Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-03-17T19:27:25.5954190Z Traceback (most recent call last): 2022-03-17T19:27:25.5955606Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:27:25.5956712Z self._pc.join(-1) 2022-03-17T19:27:25.5957473Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-03-17T19:27:25.5958340Z return _mock_self._mock_call(*args, **kwargs) 2022-03-17T19:27:25.5959186Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-03-17T19:27:25.5959943Z raise effect 2022-03-17T19:27:25.5961302Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-03-17T19:27:25.5962344Z ok (0.008s) 2022-03-17T19:27:29.0736528Z test_pcontext_wait (__main__.StartProcessesTest) ... ok (3.477s) 2022-03-17T19:27:29.0908429Z test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 3541 closing signal SIGTERM 2022-03-17T19:27:29.0931010Z ok (0.019s) 2022-03-17T19:27:29.0997095Z test_to_map (__main__.StartProcessesTest) ... ok (0.006s) 2022-03-17T19:27:29.1027150Z test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.003s) 2022-03-17T19:27:30.0671180Z test_void_function (__main__.StartProcessesTest) ... world 2022-03-17T19:27:30.1076170Z hello 2022-03-17T19:27:31.6331991Z Closing process 3543 via signal SIGTERM 2022-03-17T19:27:31.6714224Z ok (2.568s) 2022-03-17T19:27:31.6747727Z test_from_str_bad_input (__main__.StdTest) ... ok (0.003s) 2022-03-17T19:27:31.6771254Z test_from_value (__main__.StdTest) ... ok (0.002s) 2022-03-17T19:27:31.6792106Z test_from_value_map (__main__.StdTest) ... ok (0.002s) 2022-03-17T19:27:31.6792865Z 2022-03-17T19:27:31.6793971Z ---------------------------------------------------------------------- 2022-03-17T19:27:31.6794744Z Ran 38 tests in 28.861s 2022-03-17T19:27:31.6795087Z 2022-03-17T19:27:31.6795269Z OK 2022-03-17T19:27:31.6795559Z 2022-03-17T19:27:31.6795826Z Generating XML reports... 2022-03-17T19:27:31.6856715Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220317192702.xml 2022-03-17T19:27:31.6880284Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220317192702.xml 2022-03-17T19:27:31.6904309Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220317192702.xml 2022-03-17T19:27:31.6907201Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220317192702.xml 2022-03-17T19:27:33.2767892Z Running distributed/elastic/timer/api_test ... [2022-03-17 19:27:33.276134] 2022-03-17T19:27:33.2769687Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:33.276334] 2022-03-17T19:27:35.6746682Z Running distributed/elastic/timer/local_timer_example ... [2022-03-17 19:27:35.674157] 2022-03-17T19:27:35.6747567Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:35.674314] 2022-03-17T19:27:36.5099906Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example 2022-03-17T19:27:36.5111700Z 2022-03-17T19:27:36.5112401Z Running tests... 2022-03-17T19:27:36.5113807Z ---------------------------------------------------------------------- 2022-03-17T19:27:37.0971042Z test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-03-17 19:27:37,096 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-03-17T19:27:37.0972819Z [INFO] 2022-03-17 19:27:37,096 api: Starting watchdog thread... 2022-03-17T19:27:38.2128379Z [INFO] 2022-03-17 19:27:38,212 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.2151576Z [INFO] 2022-03-17 19:27:38,214 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.2297917Z [INFO] 2022-03-17 19:27:38,229 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.2414673Z [INFO] 2022-03-17 19:27:38,240 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.2423313Z [INFO] 2022-03-17 19:27:38,241 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.2887823Z [INFO] 2022-03-17 19:27:38,287 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.3089007Z [INFO] 2022-03-17 19:27:38,307 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:38.3128318Z [INFO] 2022-03-17 19:27:38,309 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:39.2958365Z [INFO] 2022-03-17 19:27:39,294 api: Reaping worker_id=[3815]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:39.2959789Z [INFO] 2022-03-17 19:27:39,294 api: Successfully reaped worker=[3815] 2022-03-17T19:27:39.3267685Z [INFO] 2022-03-17 19:27:39,325 api: Reaping worker_id=[3811]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:39.3271261Z [INFO] 2022-03-17 19:27:39,325 api: Successfully reaped worker=[3811] 2022-03-17T19:27:39.3273256Z [INFO] 2022-03-17 19:27:39,326 api: Reaping worker_id=[3813]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:39.3275231Z [INFO] 2022-03-17 19:27:39,327 api: Successfully reaped worker=[3813] 2022-03-17T19:27:39.3797248Z [INFO] 2022-03-17 19:27:39,378 api: Reaping worker_id=[3817]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:39.3799750Z [INFO] 2022-03-17 19:27:39,379 api: Successfully reaped worker=[3817] 2022-03-17T19:27:40.5687737Z [INFO] 2022-03-17 19:27:40,567 api: Stopping LocalTimerServer 2022-03-17T19:27:40.5688900Z [INFO] 2022-03-17 19:27:40,568 api: Stopping watchdog thread... 2022-03-17T19:27:40.5762673Z ok (4.065s) 2022-03-17T19:27:40.5800612Z test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-03-17 19:27:40,579 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-03-17T19:27:40.5802220Z [INFO] 2022-03-17 19:27:40,579 api: Starting watchdog thread... 2022-03-17T19:27:41.5727616Z [INFO] 2022-03-17 19:27:41,572 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.6457787Z [INFO] 2022-03-17 19:27:41,645 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.6527320Z [INFO] 2022-03-17 19:27:41,652 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.7039718Z [INFO] 2022-03-17 19:27:41,703 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.7095323Z [INFO] 2022-03-17 19:27:41,709 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.7123782Z [INFO] 2022-03-17 19:27:41,711 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.7503399Z [INFO] 2022-03-17 19:27:41,749 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:41.8077968Z [INFO] 2022-03-17 19:27:41,807 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:45.9518990Z [INFO] 2022-03-17 19:27:45,950 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:45.9722537Z [INFO] 2022-03-17 19:27:45,971 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:45.9993771Z [INFO] 2022-03-17 19:27:45,998 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:46.0043270Z [INFO] 2022-03-17 19:27:46,003 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:46.0196670Z [INFO] 2022-03-17 19:27:46,019 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:46.0267263Z [INFO] 2022-03-17 19:27:46,025 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:46.0343149Z [INFO] 2022-03-17 19:27:46,033 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:46.0737006Z [INFO] 2022-03-17 19:27:46,073 api: Timer client configured to: LocalTimerClient 2022-03-17T19:27:47.0236606Z [INFO] 2022-03-17 19:27:47,022 api: Reaping worker_id=[4906]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.0238323Z [INFO] 2022-03-17 19:27:47,022 api: Successfully reaped worker=[4906] 2022-03-17T19:27:47.0440737Z [INFO] 2022-03-17 19:27:47,043 api: Reaping worker_id=[4901]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.0442121Z [INFO] 2022-03-17 19:27:47,043 api: Successfully reaped worker=[4901] 2022-03-17T19:27:47.0752717Z [INFO] 2022-03-17 19:27:47,074 api: Reaping worker_id=[4904]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.0754436Z [INFO] 2022-03-17 19:27:47,074 api: Successfully reaped worker=[4904] 2022-03-17T19:27:47.0859174Z [INFO] 2022-03-17 19:27:47,085 api: Reaping worker_id=[4903]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.0860990Z [INFO] 2022-03-17 19:27:47,085 api: Successfully reaped worker=[4903] 2022-03-17T19:27:47.0970910Z [INFO] 2022-03-17 19:27:47,095 api: Reaping worker_id=[4907]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.0975400Z [INFO] 2022-03-17 19:27:47,096 api: Successfully reaped worker=[4907] 2022-03-17T19:27:47.1083162Z [INFO] 2022-03-17 19:27:47,107 api: Reaping worker_id=[4900]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.1085150Z [INFO] 2022-03-17 19:27:47,107 local_timer: Process with pid=4900 does not exist. Skipping 2022-03-17T19:27:47.1086789Z [INFO] 2022-03-17 19:27:47,107 api: Successfully reaped worker=[4900] 2022-03-17T19:27:47.1089021Z [INFO] 2022-03-17 19:27:47,107 api: Reaping worker_id=[4905]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.1090953Z [INFO] 2022-03-17 19:27:47,108 api: Successfully reaped worker=[4905] 2022-03-17T19:27:47.1400800Z [INFO] 2022-03-17 19:27:47,139 api: Reaping worker_id=[4902]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:27:47.1402826Z [INFO] 2022-03-17 19:27:47,139 local_timer: Process with pid=4902 does not exist. Skipping 2022-03-17T19:27:47.1404097Z [INFO] 2022-03-17 19:27:47,139 api: Successfully reaped worker=[4902] 2022-03-17T19:27:47.1472806Z [INFO] 2022-03-17 19:27:47,146 api: Stopping LocalTimerServer 2022-03-17T19:27:47.1474002Z [INFO] 2022-03-17 19:27:47,146 api: Stopping watchdog thread... 2022-03-17T19:27:47.1512087Z ok (6.575s) 2022-03-17T19:27:47.1512724Z 2022-03-17T19:27:47.1513646Z ---------------------------------------------------------------------- 2022-03-17T19:27:47.1514486Z Ran 2 tests in 10.640s 2022-03-17T19:27:47.1514856Z 2022-03-17T19:27:47.1515056Z OK 2022-03-17T19:27:47.1517003Z 2022-03-17T19:27:47.1517688Z Generating XML reports... 2022-03-17T19:27:47.1583682Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220317192736.xml 2022-03-17T19:27:48.8318345Z Running distributed/elastic/timer/local_timer_test ... [2022-03-17 19:27:48.830868] 2022-03-17T19:27:48.8320207Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:48.831025] 2022-03-17T19:27:49.7315351Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-03-17T19:27:49.7331802Z 2022-03-17T19:27:49.7332577Z Running tests... 2022-03-17T19:27:49.7333787Z ---------------------------------------------------------------------- 2022-03-17T19:27:49.7343687Z test_acquire_release (__main__.LocalTimerServerTest) 2022-03-17T19:27:50.3179862Z tests that: ... ok (0.585s) 2022-03-17T19:27:50.3191219Z test_expired_timers (__main__.LocalTimerServerTest) 2022-03-17T19:27:50.3213117Z tests that a single expired timer on a process should terminate ... ok (0.003s) 2022-03-17T19:27:50.3229977Z test_valid_timers (__main__.LocalTimerServerTest) 2022-03-17T19:27:50.3248274Z tests that valid timers are processed correctly and the process is left alone ... ok (0.003s) 2022-03-17T19:27:50.3260290Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-03-17T19:27:50.4294818Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.104s) 2022-03-17T19:27:50.4295858Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-03-17T19:27:50.4403709Z checks that the watchdog can run on an empty queue ... ok (0.011s) 2022-03-17T19:27:50.4500762Z test_client_interaction (__main__.LocalTimerTest) ... ok (0.009s) 2022-03-17T19:27:50.4623337Z test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s) 2022-03-17T19:27:50.4643432Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-03-17T19:27:51.7737067Z If a function acquires a countdown timer with default scope, ... ok (1.311s) 2022-03-17T19:27:51.8796135Z test_happy_path (__main__.LocalTimerTest) ... ok (0.106s) 2022-03-17T19:27:51.8926820Z test_no_client (__main__.LocalTimerTest) ... ok (0.013s) 2022-03-17T19:27:52.0284606Z test_timer (__main__.LocalTimerTest) ... ok (0.136s) 2022-03-17T19:27:52.0642805Z test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.030s) 2022-03-17T19:27:52.0670376Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-03-17T19:27:52.5804151Z Tests slow producer. ... ok (0.516s) 2022-03-17T19:27:52.5830960Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-03-17T19:27:53.4949639Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.914s) 2022-03-17T19:27:53.4951192Z 2022-03-17T19:27:53.4952075Z ---------------------------------------------------------------------- 2022-03-17T19:27:53.4952904Z Ran 14 tests in 3.762s 2022-03-17T19:27:53.4953277Z 2022-03-17T19:27:53.4953477Z OK 2022-03-17T19:27:53.4953775Z 2022-03-17T19:27:53.4954053Z Generating XML reports... 2022-03-17T19:27:53.5141329Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220317192749.xml 2022-03-17T19:27:53.5146259Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220317192749.xml 2022-03-17T19:27:53.5150901Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220317192749.xml 2022-03-17T19:27:55.1601536Z Running distributed/elastic/utils/distributed_test ... [2022-03-17 19:27:55.159090] 2022-03-17T19:27:55.1603588Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:55.159244] 2022-03-17T19:27:56.0527519Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test 2022-03-17T19:27:56.0542223Z 2022-03-17T19:27:56.0542721Z Running tests... 2022-03-17T19:27:56.0543899Z ---------------------------------------------------------------------- 2022-03-17T19:27:56.6598672Z test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.605s) 2022-03-17T19:27:56.6619060Z test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.002s) 2022-03-17T19:27:56.6629852Z test_create_store_single_server (__main__.DistributedUtilTest) ... skip (0.001s) 2022-03-17T19:27:59.6966804Z test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.033s) 2022-03-17T19:27:59.6975084Z test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (38dccec0eac5, 0). 2022-03-17T19:27:59.6976195Z ok (0.001s) 2022-03-17T19:27:59.6992900Z test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:38137 (errno: 98 - Address already in use). 2022-03-17T19:27:59.7054125Z [W socket.cpp:401] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use). 2022-03-17T19:27:59.7055345Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address. 2022-03-17T19:27:59.7082399Z ok (0.010s) 2022-03-17T19:27:59.7143165Z test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (38dccec0eac5, 45143). 2022-03-17T19:27:59.7144328Z ok (0.005s) 2022-03-17T19:27:59.7144653Z 2022-03-17T19:27:59.7145448Z ---------------------------------------------------------------------- 2022-03-17T19:27:59.7146226Z Ran 7 tests in 3.660s 2022-03-17T19:27:59.7146603Z 2022-03-17T19:27:59.7146813Z OK (skipped=1) 2022-03-17T19:27:59.7147173Z 2022-03-17T19:27:59.7147459Z Generating XML reports... 2022-03-17T19:27:59.7205383Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220317192756.xml 2022-03-17T19:28:01.2524221Z Running distributed/elastic/utils/logging_test ... [2022-03-17 19:28:01.251632] 2022-03-17T19:28:01.2526033Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:01.251759] 2022-03-17T19:28:02.1412417Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test 2022-03-17T19:28:02.1429961Z 2022-03-17T19:28:02.1430423Z Running tests... 2022-03-17T19:28:02.1431589Z ---------------------------------------------------------------------- 2022-03-17T19:28:02.7455001Z test_derive_module_name (__main__.LoggingTest) ... ok (0.602s) 2022-03-17T19:28:02.7478976Z test_logger_name (__main__.LoggingTest) ... ok (0.002s) 2022-03-17T19:28:02.7479518Z 2022-03-17T19:28:02.7480355Z ---------------------------------------------------------------------- 2022-03-17T19:28:02.7481123Z Ran 2 tests in 0.605s 2022-03-17T19:28:02.7481489Z 2022-03-17T19:28:02.7481677Z OK 2022-03-17T19:28:02.7482165Z 2022-03-17T19:28:02.7482429Z Generating XML reports... 2022-03-17T19:28:02.7513991Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220317192802.xml 2022-03-17T19:28:04.3145633Z Running distributed/elastic/utils/util_test ... [2022-03-17 19:28:04.313806] 2022-03-17T19:28:04.3147773Z Executing ['/opt/conda/bin/python3.7', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:04.313962] 2022-03-17T19:28:05.1960518Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-03-17T19:28:05.1982208Z 2022-03-17T19:28:05.1982542Z Running tests... 2022-03-17T19:28:05.1983712Z ---------------------------------------------------------------------- 2022-03-17T19:28:05.7776270Z test_get_data (__main__.StoreUtilTest) ... ok (0.579s) 2022-03-17T19:28:05.7794575Z test_synchronize (__main__.StoreUtilTest) ... ok (0.002s) 2022-03-17T19:28:05.8465706Z test_get_logger (__main__.UtilTest) ... ok (0.067s) 2022-03-17T19:28:05.8468716Z test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s) 2022-03-17T19:28:05.8478501Z test_get_logger_different (__main__.UtilTest) ... ok (0.001s) 2022-03-17T19:28:05.8491408Z test_get_logger_none (__main__.UtilTest) ... ok (0.001s) 2022-03-17T19:28:05.8491945Z 2022-03-17T19:28:05.8492752Z ---------------------------------------------------------------------- 2022-03-17T19:28:05.8493541Z Ran 6 tests in 0.651s 2022-03-17T19:28:05.8493907Z 2022-03-17T19:28:05.8494075Z OK 2022-03-17T19:28:05.8494369Z 2022-03-17T19:28:05.8494649Z Generating XML reports... 2022-03-17T19:28:05.8525236Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220317192805.xml 2022-03-17T19:28:05.8529337Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220317192805.xml 2022-03-17T19:28:07.4854205Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-03-17 19:28:07.484424] 2022-03-17T19:28:07.4856163Z Executing ['/opt/conda/bin/python3.7', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:07.484580] 2022-03-17T19:28:08.5564169Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer 2022-03-17T19:28:08.5592326Z 2022-03-17T19:28:08.5592801Z Running tests... 2022-03-17T19:28:08.5593971Z ---------------------------------------------------------------------- 2022-03-17T19:28:08.5618717Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:09.1595730Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip (0.600s) 2022-03-17T19:28:09.1621075Z test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:09.1712960Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5880 2022-03-17T19:28:09.1762550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5881 2022-03-17T19:28:10.3073754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:10.3257033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:12.2842857Z skip (3.124s) 2022-03-17T19:28:12.2882037Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:12.2957642Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6016 2022-03-17T19:28:12.3003702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6017 2022-03-17T19:28:13.4152654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:13.4309097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:15.4072799Z skip (3.123s) 2022-03-17T19:28:15.4113967Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:15.4176872Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6152 2022-03-17T19:28:15.4228180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6153 2022-03-17T19:28:16.5487290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:16.5802062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:18.4297706Z skip (3.022s) 2022-03-17T19:28:18.4337630Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:18.4404046Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6288 2022-03-17T19:28:18.4463721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6289 2022-03-17T19:28:19.5967613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:19.6280927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:21.7537966Z skip (3.324s) 2022-03-17T19:28:21.7573807Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:21.7632015Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6424 2022-03-17T19:28:21.7677878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6425 2022-03-17T19:28:22.8828668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:22.9208271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:24.8752081Z skip (3.121s) 2022-03-17T19:28:24.8790377Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:24.8870230Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6560 2022-03-17T19:28:24.8913895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6561 2022-03-17T19:28:26.0514560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:26.0595787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:27.9989657Z skip (3.123s) 2022-03-17T19:28:28.0029283Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:28.0108080Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6696 2022-03-17T19:28:28.0151335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6697 2022-03-17T19:28:29.1275070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:29.1435726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:31.1226293Z skip (3.124s) 2022-03-17T19:28:31.1267193Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:31.1348149Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6832 2022-03-17T19:28:31.1397064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6833 2022-03-17T19:28:32.2579597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:32.2751776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:34.5476186Z skip (3.425s) 2022-03-17T19:28:34.5520592Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:34.5599190Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6968 2022-03-17T19:28:34.5643916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6969 2022-03-17T19:28:35.7174846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:35.7506069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:38.0726080Z skip (3.525s) 2022-03-17T19:28:38.0764326Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:38.0840060Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7104 2022-03-17T19:28:38.0885564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7105 2022-03-17T19:28:39.2393502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:39.2728247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:41.2964036Z skip (3.223s) 2022-03-17T19:28:41.3003908Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:41.3080040Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7240 2022-03-17T19:28:41.3125966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7241 2022-03-17T19:28:42.4860365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:42.5100156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:44.5202930Z skip (3.224s) 2022-03-17T19:28:44.5240314Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:44.5316169Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7376 2022-03-17T19:28:44.5364066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7377 2022-03-17T19:28:45.6804780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:45.7152140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:47.6441079Z skip (3.124s) 2022-03-17T19:28:47.6479665Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:47.6555969Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7512 2022-03-17T19:28:47.6601693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7513 2022-03-17T19:28:48.7618366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:48.7818790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:50.7683415Z skip (3.124s) 2022-03-17T19:28:50.7723746Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:50.7794853Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7648 2022-03-17T19:28:50.7846638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7649 2022-03-17T19:28:51.8740132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:51.9003238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:53.9919914Z skip (3.223s) 2022-03-17T19:28:53.9958402Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:54.0033504Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7784 2022-03-17T19:28:54.0083032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7785 2022-03-17T19:28:55.1402179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:55.1521690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:57.0160043Z skip (3.024s) 2022-03-17T19:28:57.0198621Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:28:57.0272620Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7920 2022-03-17T19:28:57.0317251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7921 2022-03-17T19:28:58.1910550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:58.1987896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:00.0395430Z skip (3.023s) 2022-03-17T19:29:00.0430557Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:29:00.0509237Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8056 2022-03-17T19:29:00.0556800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8057 2022-03-17T19:29:01.1910929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:01.2191372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:03.1632907Z skip (3.124s) 2022-03-17T19:29:03.1746516Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:29:03.1796919Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8192 2022-03-17T19:29:03.1843728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8193 2022-03-17T19:29:04.2415459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:04.2416989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:04.2648359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:04.2650357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:04.2652816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:04.2727365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:04.7091403Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:04.7476554Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:14.1422897Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:14.3913911Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:14.7585119Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.7590470Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.7798537Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.7807990Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.7984449Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.7986610Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8159241Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8203548Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8376821Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8379054Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8551896Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8580075Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8780404Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.8782799Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.9203901Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:14.9206292Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:17.6182714Z ok (14.453s) 2022-03-17T19:29:17.6278971Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:29:17.6320221Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8341 2022-03-17T19:29:17.6362714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8342 2022-03-17T19:29:18.7557095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:18.7558683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:18.7985171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:18.7986793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:18.7989287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:18.8072210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:19.2343118Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:19.2773090Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:28.6842351Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:28.7333152Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:29.0938417Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.0940755Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1123720Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1131081Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1334684Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1336958Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1545637Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1555476Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1762099Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1764494Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1942904Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.1962031Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.2169775Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.2172008Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.2554264Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:29.2556462Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:31.9652111Z ok (14.346s) 2022-03-17T19:29:31.9760387Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:29:31.9806597Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8490 2022-03-17T19:29:31.9849522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8491 2022-03-17T19:29:33.0940537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:33.0942009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:33.1093861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:33.1096178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:33.1098533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:33.1150928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:33.5527016Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:33.5992228Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:43.0325931Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:43.0815701Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:43.4435218Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.4440063Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.4634494Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.4653405Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.4861948Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.4864006Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5075505Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5077807Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5287983Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5290020Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5501707Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5509234Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5733734Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.5735763Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.6190558Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:43.6192584Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:46.4183606Z ok (14.453s) 2022-03-17T19:29:46.4279677Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:29:46.4323027Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8639 2022-03-17T19:29:46.4367471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8640 2022-03-17T19:29:47.5518497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:47.5519809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:47.5750074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:47.5751873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:47.5754002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:47.5826796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:48.0149973Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:48.0556414Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:29:57.5266072Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:57.6452716Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:29:58.0010975Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0013126Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0195283Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0200987Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0427352Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0429880Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0611610Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0645643Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0833573Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.0835578Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.1007226Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.1053659Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.1224113Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.1226122Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.1631763Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:29:58.1633875Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:00.8675748Z ok (14.449s) 2022-03-17T19:30:00.8784827Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:00.8831137Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8788 2022-03-17T19:30:00.8873280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8789 2022-03-17T19:30:02.0079952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:02.0081237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:02.0254425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:02.0255781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:02.0257857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:02.0287235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:02.4659668Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:02.5094775Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:11.8160749Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:30:11.9426881Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:30:12.2436974Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.2438910Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.2624064Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.2644303Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.2848049Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.2850196Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3036348Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3061527Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3252835Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3255404Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3443515Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3456199Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3643782Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3645914Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3863410Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:12.3865497Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:15.1165183Z ok (14.249s) 2022-03-17T19:30:15.1277923Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:15.1318954Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8937 2022-03-17T19:30:15.1363492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8938 2022-03-17T19:30:16.2955160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:16.2956452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:16.3082188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:16.3084638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:16.3087091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:16.3163770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:16.7424423Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:16.7884057Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:26.2339235Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:30:26.3424547Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:30:26.6444122Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.6446179Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.6618283Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.6630737Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.6835406Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.6837363Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7026638Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7039487Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7233860Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7235872Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7441661Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7443785Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7607832Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7609814Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7855269Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:26.7857249Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:30:29.6686889Z ok (14.552s) 2022-03-17T19:30:29.6723378Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:29.6793910Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9086 2022-03-17T19:30:29.6838043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9087 2022-03-17T19:30:30.8134545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:30.8135935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:30.8164733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:30.8166008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:30.8168513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:30.8241746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:31.2252318Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:31.2772141Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:33.5934203Z ok (3.924s) 2022-03-17T19:30:33.6017040Z test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:33.6063985Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9235 2022-03-17T19:30:33.6270795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9236 2022-03-17T19:30:34.7801755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:34.7803598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:34.8010348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:34.8012052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:34.8014260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:34.8117433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:44.4911648Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:44.5305728Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:30:47.3576663Z ok (13.764s) 2022-03-17T19:30:47.3657074Z test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:47.3699429Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9384 2022-03-17T19:30:47.3746138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9385 2022-03-17T19:30:48.4992009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:48.4993791Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-03-17T19:30:48.5027226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:48.5028809Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-03-17T19:30:50.4822631Z ok (3.124s) 2022-03-17T19:30:50.4847516Z test_sharding (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:50.4850893Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip (0.003s) 2022-03-17T19:30:50.4901741Z test_step (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:50.4952565Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9520 2022-03-17T19:30:50.4999982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9521 2022-03-17T19:30:51.6665955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:51.6708594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:53.5072409Z skip (3.022s) 2022-03-17T19:30:53.5137497Z test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:53.5207570Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9656 2022-03-17T19:30:53.5251851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9657 2022-03-17T19:30:54.6472972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:54.6557994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:56.9334256Z skip (3.426s) 2022-03-17T19:30:56.9345193Z test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:30:56.9429053Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9792 2022-03-17T19:30:56.9474919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9793 2022-03-17T19:30:58.0800467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:58.1201797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:58.1324221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:58.1326408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:58.1327770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:58.1329518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:58.1729148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:30:58.1730984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:31:00.1550993Z ok (3.221s) 2022-03-17T19:31:00.1559024Z test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:31:00.1646748Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10000 2022-03-17T19:31:00.1706317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10001 2022-03-17T19:31:01.3185679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:01.3187797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:01.3388494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:01.3400906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:01.3403490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:01.3499277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:01.7219391Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:01.7832295Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:11.1902855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:31:11.1904126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:31:14.1984805Z ok (14.043s) 2022-03-17T19:31:14.2000709Z test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:31:14.2066609Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10149 2022-03-17T19:31:14.2122785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10150 2022-03-17T19:31:15.3368964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:15.3380115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:17.3197530Z skip (3.121s) 2022-03-17T19:31:17.3215646Z test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:31:17.3299973Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10285 2022-03-17T19:31:17.3349565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10286 2022-03-17T19:31:18.4672316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:18.4794240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:20.4433524Z skip (3.123s) 2022-03-17T19:31:20.4498536Z test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:20.4561220Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10421 2022-03-17T19:31:21.5315879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:21.5317752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:21.5319999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:23.4627006Z ok (3.019s) 2022-03-17T19:31:23.4666076Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:23.4740612Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10490 2022-03-17T19:31:24.5900131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:24.5902225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:24.5904364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:24.9858258Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:26.8818293Z ok (3.419s) 2022-03-17T19:31:26.8843150Z test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:26.8913049Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10565 2022-03-17T19:31:27.9504571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:27.9505867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:27.9508070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:29.7978435Z ok (2.916s) 2022-03-17T19:31:29.8069511Z test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:29.8117149Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10634 2022-03-17T19:31:30.8800486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:30.8802763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:30.8804960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:31.2759246Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:33.1196186Z ok (3.321s) 2022-03-17T19:31:33.1225128Z test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:33.1300478Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10709 2022-03-17T19:31:34.1963563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:34.1964989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:34.1967237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:34.5922837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:36.4377935Z ok (3.318s) 2022-03-17T19:31:36.4413872Z test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:36.4479867Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10784 2022-03-17T19:31:37.5480275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:37.5481688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:37.5484269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:37.9448428Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:39.8552623Z ok (3.417s) 2022-03-17T19:31:39.8579344Z test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:39.8649318Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10859 2022-03-17T19:31:40.9293002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:40.9294467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:40.9296570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:41.3180292Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:31:43.1722961Z ok (3.317s) 2022-03-17T19:31:43.1749498Z test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:31:43.1825428Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10934 2022-03-17T19:31:44.3067705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:44.3068920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:44.3070961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:46.2897173Z ok (3.117s) 2022-03-17T19:31:46.2897689Z 2022-03-17T19:31:46.2903252Z ---------------------------------------------------------------------- 2022-03-17T19:31:46.2904292Z Ran 42 tests in 217.730s 2022-03-17T19:31:46.2904715Z 2022-03-17T19:31:46.2905024Z OK (skipped=23) 2022-03-17T19:31:46.2905427Z 2022-03-17T19:31:46.2905720Z Generating XML reports... 2022-03-17T19:31:46.3049703Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220317192808.xml 2022-03-17T19:31:46.3060425Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220317192808.xml 2022-03-17T19:31:47.9007082Z Running distributed/pipeline/sync/skip/test_api ... [2022-03-17 19:31:47.899741] 2022-03-17T19:31:47.9009650Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-03-17 19:31:47.899874] 2022-03-17T19:31:48.9099465Z ============================= test session starts ============================== 2022-03-17T19:31:48.9100877Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:31:48.9117870Z cachedir: .pytest_cache 2022-03-17T19:31:48.9119531Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:31:48.9120657Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:31:48.9121460Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:31:48.9122841Z plugins: hypothesis-4.53.2 2022-03-17T19:31:48.9254041Z collecting ...  2022-03-17T19:31:48.9255417Z collected 3 items  2022-03-17T19:31:48.9257828Z 2022-03-17T19:31:48.9282607Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-03-17T19:31:48.9295289Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-03-17T19:31:48.9348393Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-03-17T19:31:48.9350111Z 2022-03-17T19:31:48.9351328Z ============================== 3 passed in 0.03s =============================== 2022-03-17T19:31:50.6103494Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-03-17 19:31:50.609675] 2022-03-17T19:31:50.6105301Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-03-17 19:31:50.609861] 2022-03-17T19:31:51.5443160Z ============================= test session starts ============================== 2022-03-17T19:31:51.5444597Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:31:51.5462334Z cachedir: .pytest_cache 2022-03-17T19:31:51.5464358Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:31:51.5465427Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:31:51.5466172Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:31:51.5467104Z plugins: hypothesis-4.53.2 2022-03-17T19:31:51.6243921Z collecting ...  2022-03-17T19:31:51.6245208Z collected 13 items  2022-03-17T19:31:51.6246571Z 2022-03-17T19:31:51.6357209Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:02.8384385Z PASSED [ 7%] 2022-03-17T19:32:02.8435947Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:03.5158544Z PASSED [ 15%] 2022-03-17T19:32:03.5190109Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:03.6501079Z PASSED [ 23%] 2022-03-17T19:32:03.6534561Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:03.6701457Z SKIPPED [ 30%] 2022-03-17T19:32:03.6754859Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:03.8317595Z PASSED [ 38%] 2022-03-17T19:32:03.8355152Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:03.9734766Z PASSED [ 46%] 2022-03-17T19:32:03.9771251Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.1048555Z PASSED [ 53%] 2022-03-17T19:32:04.1085070Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.1231299Z SKIPPED [ 61%] 2022-03-17T19:32:04.1276912Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.2644146Z PASSED [ 69%] 2022-03-17T19:32:04.2698022Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.4087851Z PASSED [ 76%] 2022-03-17T19:32:04.4122293Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.5397766Z PASSED [ 84%] 2022-03-17T19:32:04.5435876Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.5574243Z SKIPPED [ 92%] 2022-03-17T19:32:04.5608072Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:04.5904324Z PASSED [100%] 2022-03-17T19:32:04.5906768Z 2022-03-17T19:32:04.5907384Z =========================== short test summary info ============================ 2022-03-17T19:32:04.5909099Z SKIPPED [3] distributed/pipeline/sync/skip/test_gpipe.py:24: at least 3 cuda devices required 2022-03-17T19:32:04.5910758Z ======================== 10 passed, 3 skipped in 13.05s ======================== 2022-03-17T19:32:07.8188880Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-03-17 19:32:07.817993] 2022-03-17T19:32:07.8190760Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-03-17 19:32:07.818174] 2022-03-17T19:32:08.7941757Z ============================= test session starts ============================== 2022-03-17T19:32:08.7942366Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:08.7961512Z cachedir: .pytest_cache 2022-03-17T19:32:08.7962563Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:08.7963169Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:08.7963678Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:08.7967966Z plugins: hypothesis-4.53.2 2022-03-17T19:32:08.8116377Z collecting ...  2022-03-17T19:32:08.8117615Z collected 6 items  2022-03-17T19:32:08.8118426Z 2022-03-17T19:32:08.8148320Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-03-17T19:32:08.8170254Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-03-17T19:32:08.8184181Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-03-17T19:32:08.8198436Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-03-17T19:32:08.8214792Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-03-17T19:32:08.8233441Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-03-17T19:32:08.8234175Z 2022-03-17T19:32:08.8234928Z ============================== 6 passed in 0.03s =============================== 2022-03-17T19:32:10.6461743Z Running distributed/pipeline/sync/skip/test_leak ... [2022-03-17 19:32:10.645324] 2022-03-17T19:32:10.6463465Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-03-17 19:32:10.645501] 2022-03-17T19:32:11.5812211Z ============================= test session starts ============================== 2022-03-17T19:32:11.5813513Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:11.5833467Z cachedir: .pytest_cache 2022-03-17T19:32:11.5835139Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:11.5836216Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:11.5836971Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:11.5837896Z plugins: hypothesis-4.53.2 2022-03-17T19:32:11.5989177Z collecting ...  2022-03-17T19:32:11.5990254Z collected 8 items  2022-03-17T19:32:11.5990740Z 2022-03-17T19:32:11.6721321Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.7174056Z PASSED [ 12%] 2022-03-17T19:32:11.7206195Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.7398205Z PASSED [ 25%] 2022-03-17T19:32:11.7437516Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.7663913Z PASSED [ 37%] 2022-03-17T19:32:11.7713698Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.7986535Z PASSED [ 50%] 2022-03-17T19:32:11.8035531Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.8243415Z PASSED [ 62%] 2022-03-17T19:32:11.8288145Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.8523991Z PASSED [ 75%] 2022-03-17T19:32:11.8574749Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.8718014Z PASSED [ 87%] 2022-03-17T19:32:11.8750862Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:11.8910752Z PASSED [100%] 2022-03-17T19:32:11.8911176Z 2022-03-17T19:32:11.8911964Z ============================== 8 passed in 0.31s =============================== 2022-03-17T19:32:13.5526625Z Running distributed/pipeline/sync/skip/test_portal ... [2022-03-17 19:32:13.551784] 2022-03-17T19:32:13.5528350Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-03-17 19:32:13.551959] 2022-03-17T19:32:14.5697992Z ============================= test session starts ============================== 2022-03-17T19:32:14.5699430Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:14.5719949Z cachedir: .pytest_cache 2022-03-17T19:32:14.5722132Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:14.5723426Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:14.5724235Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:14.5725866Z plugins: hypothesis-4.53.2 2022-03-17T19:32:14.6703364Z collecting ...  2022-03-17T19:32:14.6704667Z collected 10 items  2022-03-17T19:32:14.6705305Z 2022-03-17T19:32:15.0172340Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%] 2022-03-17T19:32:15.0192943Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-03-17T19:32:15.0205697Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-03-17T19:32:15.0214909Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-03-17T19:32:15.0227090Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-03-17T19:32:15.0237326Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-03-17T19:32:15.0249353Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-03-17T19:32:15.0260198Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-03-17T19:32:15.0273331Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-03-17T19:32:15.0290749Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-03-17T19:32:15.0291428Z 2022-03-17T19:32:15.0292083Z ============================== 10 passed in 0.46s ============================== 2022-03-17T19:32:16.5752689Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-03-17 19:32:16.574428] 2022-03-17T19:32:16.5754806Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-03-17 19:32:16.574609] 2022-03-17T19:32:17.5133162Z ============================= test session starts ============================== 2022-03-17T19:32:17.5133788Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:17.5153969Z cachedir: .pytest_cache 2022-03-17T19:32:17.5154610Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:17.5155063Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:17.5155463Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:17.5155918Z plugins: hypothesis-4.53.2 2022-03-17T19:32:17.5302610Z collecting ...  2022-03-17T19:32:17.5304315Z collected 7 items  2022-03-17T19:32:17.5304872Z 2022-03-17T19:32:17.5369335Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED [ 14%] 2022-03-17T19:32:17.5386338Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED [ 28%] 2022-03-17T19:32:17.5403404Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%] 2022-03-17T19:32:17.5417474Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%] 2022-03-17T19:32:17.5433289Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%] 2022-03-17T19:32:17.5447559Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%] 2022-03-17T19:32:17.5466363Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%] 2022-03-17T19:32:17.5466637Z 2022-03-17T19:32:17.5466929Z ============================== 7 passed in 0.03s =============================== 2022-03-17T19:32:19.3446786Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-03-17 19:32:19.343762] 2022-03-17T19:32:19.3448777Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-03-17 19:32:19.343940] 2022-03-17T19:32:20.3184943Z ============================= test session starts ============================== 2022-03-17T19:32:20.3186600Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:20.3203537Z cachedir: .pytest_cache 2022-03-17T19:32:20.3205586Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:20.3206873Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:20.3207737Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:20.3208724Z plugins: hypothesis-4.53.2 2022-03-17T19:32:20.4029319Z collecting ...  2022-03-17T19:32:20.4030793Z collected 6 items  2022-03-17T19:32:20.4031277Z 2022-03-17T19:32:20.4066758Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-03-17T19:32:20.8892049Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%] 2022-03-17T19:32:20.8904649Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-03-17T19:32:20.8913290Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-03-17T19:32:20.8924701Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-03-17T19:32:20.8942330Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-03-17T19:32:20.8943105Z 2022-03-17T19:32:20.8943926Z ============================== 6 passed in 0.58s =============================== 2022-03-17T19:32:22.4001669Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-03-17 19:32:22.399387] 2022-03-17T19:32:22.4003590Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-03-17 19:32:22.399566] 2022-03-17T19:32:23.3518527Z ============================= test session starts ============================== 2022-03-17T19:32:23.3519017Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:23.3541930Z cachedir: .pytest_cache 2022-03-17T19:32:23.3542786Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:23.3543276Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:23.3543621Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:23.3544059Z plugins: hypothesis-4.53.2 2022-03-17T19:32:23.3722388Z collecting ...  2022-03-17T19:32:23.3722856Z collected 9 items  2022-03-17T19:32:23.3724082Z 2022-03-17T19:32:23.3756787Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2022-03-17T19:32:23.3772635Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2022-03-17T19:32:23.3786371Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2022-03-17T19:32:23.3802658Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2022-03-17T19:32:23.3816180Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2022-03-17T19:32:23.3833086Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2022-03-17T19:32:23.3847380Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2022-03-17T19:32:23.3862604Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2022-03-17T19:32:23.3884520Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2022-03-17T19:32:23.3885282Z 2022-03-17T19:32:23.3886021Z ============================== 9 passed in 0.04s =============================== 2022-03-17T19:32:25.1858851Z Running distributed/pipeline/sync/test_balance ... [2022-03-17 19:32:25.185218] 2022-03-17T19:32:25.1860514Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-03-17 19:32:25.185394] 2022-03-17T19:32:26.1965640Z ============================= test session starts ============================== 2022-03-17T19:32:26.1966992Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:26.1985102Z cachedir: .pytest_cache 2022-03-17T19:32:26.1987032Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:26.1988320Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:26.1989087Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:26.1990043Z plugins: hypothesis-4.53.2 2022-03-17T19:32:26.2845850Z collecting ...  2022-03-17T19:32:26.2847112Z collected 18 items  2022-03-17T19:32:26.2849958Z 2022-03-17T19:32:26.2877861Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 5%] 2022-03-17T19:32:26.2891345Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%] 2022-03-17T19:32:26.2904055Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%] 2022-03-17T19:32:26.2916484Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%] 2022-03-17T19:32:26.2925377Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%] 2022-03-17T19:32:26.2932834Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%] 2022-03-17T19:32:27.3028401Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%] 2022-03-17T19:32:27.6602121Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%] 2022-03-17T19:32:36.5658260Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%] 2022-03-17T19:32:36.5772279Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%] 2022-03-17T19:32:36.5803009Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%] 2022-03-17T19:32:36.5827603Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%] 2022-03-17T19:32:37.5871567Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%] 2022-03-17T19:32:38.6481432Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%] 2022-03-17T19:32:39.6520191Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 83%] 2022-03-17T19:32:40.6548655Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%] 2022-03-17T19:32:40.6577764Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%] 2022-03-17T19:32:40.6646558Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-03-17T19:32:40.6647443Z 2022-03-17T19:32:40.6648279Z =============================== warnings summary =============================== 2022-03-17T19:32:40.6649520Z test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] 2022-03-17T19:32:40.6650908Z test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] 2022-03-17T19:32:40.6654118Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:32:40.6656199Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:32:40.6656879Z 2022-03-17T19:32:40.6657654Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2022-03-17T19:32:40.6658640Z =========================== short test summary info ============================ 2022-03-17T19:32:40.6659612Z SKIPPED [2] distributed/pipeline/sync/test_balance.py:47: Flaky due to time.sleep() 2022-03-17T19:32:40.6661138Z ================== 16 passed, 2 skipped, 2 warnings in 14.47s ================== 2022-03-17T19:32:43.3106421Z Running distributed/pipeline/sync/test_bugs ... [2022-03-17 19:32:43.309626] 2022-03-17T19:32:43.3108495Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-03-17 19:32:43.309805] 2022-03-17T19:32:44.2452848Z ============================= test session starts ============================== 2022-03-17T19:32:44.2454125Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:44.2472501Z cachedir: .pytest_cache 2022-03-17T19:32:44.2474485Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:44.2475739Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:44.2476507Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:44.2477442Z plugins: hypothesis-4.53.2 2022-03-17T19:32:44.3221775Z collecting ...  2022-03-17T19:32:44.3223144Z collected 4 items  2022-03-17T19:32:44.3223732Z 2022-03-17T19:32:44.3331050Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:44.3777882Z PASSED [ 25%] 2022-03-17T19:32:44.3819058Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:44.3985509Z PASSED [ 50%] 2022-03-17T19:32:44.7341084Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:46.5172142Z PASSED [ 75%] 2022-03-17T19:32:46.5209004Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:32:46.6330176Z PASSED [100%] 2022-03-17T19:32:46.6331188Z 2022-03-17T19:32:46.6332396Z ============================== 4 passed in 2.39s =============================== 2022-03-17T19:32:48.2224857Z Running distributed/pipeline/sync/test_checkpoint ... [2022-03-17 19:32:48.221595] 2022-03-17T19:32:48.2226548Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-03-17 19:32:48.221773] 2022-03-17T19:32:49.1825123Z ============================= test session starts ============================== 2022-03-17T19:32:49.1826999Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:49.1843019Z cachedir: .pytest_cache 2022-03-17T19:32:49.1844963Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:49.1846158Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:49.1846923Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:49.1847861Z plugins: hypothesis-4.53.2 2022-03-17T19:32:49.2600141Z collecting ...  2022-03-17T19:32:49.2601293Z collected 9 items  2022-03-17T19:32:49.2601776Z 2022-03-17T19:32:49.2688284Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%] 2022-03-17T19:32:49.6471345Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%] 2022-03-17T19:32:49.6486027Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%] 2022-03-17T19:32:49.6498259Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%] 2022-03-17T19:32:49.6520165Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%] 2022-03-17T19:32:49.8687217Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%] 2022-03-17T19:32:49.8704988Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%] 2022-03-17T19:32:49.8713602Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%] 2022-03-17T19:32:49.8735644Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-03-17T19:32:49.8736341Z 2022-03-17T19:32:49.8737090Z ============================== 9 passed in 0.69s =============================== 2022-03-17T19:32:51.3852958Z Running distributed/pipeline/sync/test_copy ... [2022-03-17 19:32:51.384294] 2022-03-17T19:32:51.3854871Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-03-17 19:32:51.384467] 2022-03-17T19:32:52.3302042Z ============================= test session starts ============================== 2022-03-17T19:32:52.3303299Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:52.3321686Z cachedir: .pytest_cache 2022-03-17T19:32:52.3323477Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:52.3324506Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:52.3325273Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:52.3326182Z plugins: hypothesis-4.53.2 2022-03-17T19:32:52.4057508Z collecting ...  2022-03-17T19:32:52.4058524Z collected 5 items  2022-03-17T19:32:52.4061417Z 2022-03-17T19:32:52.4147799Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED [ 20%] 2022-03-17T19:32:52.7484668Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED [ 40%] 2022-03-17T19:32:52.9948194Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED [ 60%] 2022-03-17T19:32:53.3237600Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED [ 80%] 2022-03-17T19:32:53.3259346Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%] 2022-03-17T19:32:53.3260169Z 2022-03-17T19:32:53.3261144Z ============================== 5 passed in 1.00s =============================== 2022-03-17T19:32:55.5580952Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-03-17 19:32:55.557170] 2022-03-17T19:32:55.5582959Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-03-17 19:32:55.557314] 2022-03-17T19:32:56.5319786Z ============================= test session starts ============================== 2022-03-17T19:32:56.5321078Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:56.5346583Z cachedir: .pytest_cache 2022-03-17T19:32:56.5348215Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:56.5349291Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:56.5350045Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:56.5350978Z plugins: hypothesis-4.53.2 2022-03-17T19:32:56.5622696Z collecting ...  2022-03-17T19:32:56.5623774Z collected 11 items  2022-03-17T19:32:56.5624247Z 2022-03-17T19:32:56.7035394Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-03-17T19:32:56.7571038Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-03-17T19:32:56.8188266Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-03-17T19:32:56.8859418Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-03-17T19:32:56.9380295Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-03-17T19:32:56.9965258Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-03-17T19:32:56.9986058Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-03-17T19:32:57.0340908Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-03-17T19:32:57.2296986Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-03-17T19:32:57.3691891Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-03-17T19:32:57.4144394Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-03-17T19:32:57.4145168Z 2022-03-17T19:32:57.4145906Z ============================== 11 passed in 0.88s ============================== 2022-03-17T19:32:59.0171408Z Running distributed/pipeline/sync/test_dependency ... [2022-03-17 19:32:59.016102] 2022-03-17T19:32:59.0173385Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-03-17 19:32:59.016302] 2022-03-17T19:32:59.9607533Z ============================= test session starts ============================== 2022-03-17T19:32:59.9608271Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:32:59.9633031Z cachedir: .pytest_cache 2022-03-17T19:32:59.9634113Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:32:59.9634645Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:32:59.9635003Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:32:59.9635439Z plugins: hypothesis-4.53.2 2022-03-17T19:33:00.0457514Z collecting ...  2022-03-17T19:33:00.0457905Z collected 6 items  2022-03-17T19:33:00.0460087Z 2022-03-17T19:33:00.3856512Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED [ 16%] 2022-03-17T19:33:00.3867855Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2022-03-17T19:33:00.3881416Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2022-03-17T19:33:00.3893603Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2022-03-17T19:33:00.3903780Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2022-03-17T19:33:00.3920869Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2022-03-17T19:33:00.3921642Z 2022-03-17T19:33:00.3922585Z ============================== 6 passed in 0.43s =============================== 2022-03-17T19:33:02.0652338Z Running distributed/pipeline/sync/test_inplace ... [2022-03-17 19:33:02.064264] 2022-03-17T19:33:02.0654057Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-03-17 19:33:02.064441] 2022-03-17T19:33:03.0129864Z ============================= test session starts ============================== 2022-03-17T19:33:03.0131418Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:03.0153912Z cachedir: .pytest_cache 2022-03-17T19:33:03.0155998Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:03.0157123Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:03.0158354Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:03.0159324Z plugins: hypothesis-4.53.2 2022-03-17T19:33:03.0269522Z collecting ...  2022-03-17T19:33:03.0270922Z collected 3 items  2022-03-17T19:33:03.0271394Z 2022-03-17T19:33:03.1046810Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:03.1852764Z PASSED [ 33%] 2022-03-17T19:33:03.1905849Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:03.2026750Z XFAIL [ 66%] 2022-03-17T19:33:03.2075527Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:03.2336933Z XFAIL [100%] 2022-03-17T19:33:03.2338161Z 2022-03-17T19:33:03.2338367Z =========================== short test summary info ============================ 2022-03-17T19:33:03.2339046Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad 2022-03-17T19:33:03.2339684Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2022-03-17T19:33:03.2341342Z ========================= 1 passed, 2 xfailed in 0.22s ========================= 2022-03-17T19:33:04.9122752Z Running distributed/pipeline/sync/test_microbatch ... [2022-03-17 19:33:04.911211] 2022-03-17T19:33:04.9124433Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-03-17 19:33:04.911391] 2022-03-17T19:33:05.8925306Z ============================= test session starts ============================== 2022-03-17T19:33:05.8925917Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:05.8945440Z cachedir: .pytest_cache 2022-03-17T19:33:05.8946089Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:05.8946575Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:05.8946933Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:05.8947357Z plugins: hypothesis-4.53.2 2022-03-17T19:33:05.9240196Z collecting ...  2022-03-17T19:33:05.9240743Z collected 10 items  2022-03-17T19:33:05.9242259Z 2022-03-17T19:33:05.9277906Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED [ 10%] 2022-03-17T19:33:05.9292381Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%] 2022-03-17T19:33:05.9309828Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED [ 30%] 2022-03-17T19:33:05.9326159Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%] 2022-03-17T19:33:05.9341429Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%] 2022-03-17T19:33:05.9360690Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED [ 60%] 2022-03-17T19:33:05.9401451Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%] 2022-03-17T19:33:05.9418146Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED [ 80%] 2022-03-17T19:33:05.9432667Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%] 2022-03-17T19:33:05.9456689Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%] 2022-03-17T19:33:05.9457424Z 2022-03-17T19:33:05.9458165Z ============================== 10 passed in 0.05s ============================== 2022-03-17T19:33:07.7729677Z Running distributed/pipeline/sync/test_phony ... [2022-03-17 19:33:07.772206] 2022-03-17T19:33:07.7731285Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-03-17 19:33:07.772383] 2022-03-17T19:33:08.7082438Z ============================= test session starts ============================== 2022-03-17T19:33:08.7083805Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:08.7101468Z cachedir: .pytest_cache 2022-03-17T19:33:08.7103469Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:08.7104744Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:08.7105491Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:08.7106434Z plugins: hypothesis-4.53.2 2022-03-17T19:33:08.7247475Z collecting ...  2022-03-17T19:33:08.7248640Z collected 4 items  2022-03-17T19:33:08.7249182Z 2022-03-17T19:33:08.7279484Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-03-17T19:33:08.7291525Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-03-17T19:33:08.7303851Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-03-17T19:33:08.7354691Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-03-17T19:33:08.7355473Z 2022-03-17T19:33:08.7356252Z ============================== 4 passed in 0.03s =============================== 2022-03-17T19:33:10.5524377Z Running distributed/pipeline/sync/test_pipe ... [2022-03-17 19:33:10.551467] 2022-03-17T19:33:10.5526108Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-03-17 19:33:10.551640] 2022-03-17T19:33:11.4941917Z ============================= test session starts ============================== 2022-03-17T19:33:11.4942506Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:11.4963282Z cachedir: .pytest_cache 2022-03-17T19:33:11.4964402Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:11.4965070Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:11.4977286Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:11.4977692Z plugins: hypothesis-4.53.2 2022-03-17T19:33:11.6729961Z collecting ...  2022-03-17T19:33:11.6730456Z collected 56 items  2022-03-17T19:33:11.6731839Z 2022-03-17T19:33:11.6799519Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED [ 1%] 2022-03-17T19:33:11.6859593Z distributed/pipeline/sync/test_pipe.py::test_parameters libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.7271402Z PASSED [ 3%] 2022-03-17T19:33:11.7301317Z distributed/pipeline/sync/test_pipe.py::test_public_attrs libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.7459486Z PASSED [ 5%] 2022-03-17T19:33:11.7510195Z distributed/pipeline/sync/test_pipe.py::test_sequential_like libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.7761075Z PASSED [ 7%] 2022-03-17T19:33:11.7822955Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.8059016Z PASSED [ 8%] 2022-03-17T19:33:11.8111715Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.8402497Z PASSED [ 10%] 2022-03-17T19:33:11.8455946Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.8591634Z PASSED [ 12%] 2022-03-17T19:33:11.8640655Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.8938369Z PASSED [ 14%] 2022-03-17T19:33:11.8980632Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.9247488Z PASSED [ 16%] 2022-03-17T19:33:11.9309535Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.9554722Z PASSED [ 17%] 2022-03-17T19:33:11.9593508Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.9770154Z PASSED [ 19%] 2022-03-17T19:33:11.9813946Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:11.9998250Z PASSED [ 21%] 2022-03-17T19:33:12.0040774Z distributed/pipeline/sync/test_pipe.py::test_no_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.0205717Z PASSED [ 23%] 2022-03-17T19:33:12.0247859Z distributed/pipeline/sync/test_pipe.py::test_exception libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.0401055Z PASSED [ 25%] 2022-03-17T19:33:12.0444162Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.2677709Z PASSED [ 26%] 2022-03-17T19:33:12.2712284Z distributed/pipeline/sync/test_pipe.py::test_nested_input libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.2908981Z PASSED [ 28%] 2022-03-17T19:33:12.2941525Z distributed/pipeline/sync/test_pipe.py::test_input_pair libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.3106266Z PASSED [ 30%] 2022-03-17T19:33:12.3153711Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.3430767Z PASSED [ 32%] 2022-03-17T19:33:12.3474600Z distributed/pipeline/sync/test_pipe.py::test_input_singleton libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.3761342Z PASSED [ 33%] 2022-03-17T19:33:12.3798367Z distributed/pipeline/sync/test_pipe.py::test_input_varargs libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.3937108Z PASSED [ 35%] 2022-03-17T19:33:12.3969198Z distributed/pipeline/sync/test_pipe.py::test_non_tensor libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.4131616Z PASSED [ 37%] 2022-03-17T19:33:12.4191575Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.4429366Z PASSED [ 39%] 2022-03-17T19:33:12.4475574Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.4865258Z PASSED [ 41%] 2022-03-17T19:33:12.4910453Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.5214757Z PASSED [ 42%] 2022-03-17T19:33:12.5260629Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.5585709Z PASSED [ 44%] 2022-03-17T19:33:12.5617257Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.5763418Z PASSED [ 46%] 2022-03-17T19:33:12.5797023Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.5924847Z PASSED [ 48%] 2022-03-17T19:33:12.5958298Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.6122749Z PASSED [ 50%] 2022-03-17T19:33:12.6170853Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.6349053Z PASSED [ 51%] 2022-03-17T19:33:12.6395847Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.6582205Z PASSED [ 53%] 2022-03-17T19:33:12.6626613Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.6816076Z PASSED [ 55%] 2022-03-17T19:33:12.6866571Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.7146879Z PASSED [ 57%] 2022-03-17T19:33:12.7194592Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.7466302Z PASSED [ 58%] 2022-03-17T19:33:12.7523596Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.7726589Z PASSED [ 60%] 2022-03-17T19:33:12.7787339Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.8100318Z PASSED [ 62%] 2022-03-17T19:33:12.8128057Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.8392158Z PASSED [ 64%] 2022-03-17T19:33:12.8426566Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.8679720Z PASSED [ 66%] 2022-03-17T19:33:12.8724239Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.8943445Z PASSED [ 67%] 2022-03-17T19:33:12.8996630Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.9325773Z PASSED [ 69%] 2022-03-17T19:33:12.9370052Z distributed/pipeline/sync/test_pipe.py::test_devices libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.9532949Z PASSED [ 71%] 2022-03-17T19:33:12.9591975Z distributed/pipeline/sync/test_pipe.py::test_partitions libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:12.9815703Z PASSED [ 73%] 2022-03-17T19:33:12.9850381Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.4048857Z PASSED [ 75%] 2022-03-17T19:33:13.4084330Z distributed/pipeline/sync/test_pipe.py::test_deny_moving libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.4264845Z PASSED [ 76%] 2022-03-17T19:33:13.4314093Z distributed/pipeline/sync/test_pipe.py::test_empty_module libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.4436887Z PASSED [ 78%] 2022-03-17T19:33:13.4469441Z distributed/pipeline/sync/test_pipe.py::test_named_children libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.4612973Z PASSED [ 80%] 2022-03-17T19:33:13.4660317Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.4787385Z PASSED [ 82%] 2022-03-17T19:33:13.4840940Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.5094566Z PASSED [ 83%] 2022-03-17T19:33:13.5129194Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:13.5267700Z PASSED [ 85%] 2022-03-17T19:33:13.5314443Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:24.9057578Z PASSED [ 87%] 2022-03-17T19:33:24.9101230Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:24.9572711Z PASSED [ 89%] 2022-03-17T19:33:24.9605163Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:25.2825922Z PASSED [ 91%] 2022-03-17T19:33:25.2872543Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:25.3063347Z PASSED [ 92%] 2022-03-17T19:33:25.3127914Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:25.3416734Z PASSED [ 94%] 2022-03-17T19:33:25.3460123Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:25.3605645Z PASSED [ 96%] 2022-03-17T19:33:25.3652252Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:25.3812627Z PASSED [ 98%] 2022-03-17T19:33:25.3866986Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:25.4940187Z PASSED [100%] 2022-03-17T19:33:25.4940803Z 2022-03-17T19:33:25.4941694Z =============================== warnings summary =============================== 2022-03-17T19:33:25.4942913Z test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible 2022-03-17T19:33:25.4943952Z test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small 2022-03-17T19:33:25.4945627Z /opt/conda/lib/python3.7/site-packages/_pytest/python.py:192: PytestRemovedIn8Warning: Passing None has been deprecated. 2022-03-17T19:33:25.4948196Z See https://docs.pytest.org/en/latest/how-to/capture-warnings.html#additional-use-cases-of-warnings-in-tests for alternatives in common use cases. 2022-03-17T19:33:25.4949455Z result = testfunction(**testargs) 2022-03-17T19:33:25.4949943Z 2022-03-17T19:33:25.4951270Z test/distributed/pipeline/sync/test_pipe.py::test_partitions 2022-03-17T19:33:25.4954270Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:33:25.4956353Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:33:25.4957020Z 2022-03-17T19:33:25.4957775Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2022-03-17T19:33:25.4959168Z ======================= 56 passed, 3 warnings in 14.00s ======================== 2022-03-17T19:33:28.6762571Z Running distributed/pipeline/sync/test_pipeline ... [2022-03-17 19:33:28.675615] 2022-03-17T19:33:28.6764192Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-03-17 19:33:28.675736] 2022-03-17T19:33:29.6132027Z ============================= test session starts ============================== 2022-03-17T19:33:29.6133311Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:29.6151443Z cachedir: .pytest_cache 2022-03-17T19:33:29.6153061Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:29.6154107Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:29.6154888Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:29.6155832Z plugins: hypothesis-4.53.2 2022-03-17T19:33:29.6285408Z collecting ...  2022-03-17T19:33:29.6286013Z collected 1 item  2022-03-17T19:33:29.6286304Z 2022-03-17T19:33:29.6319924Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-03-17T19:33:29.6320652Z 2022-03-17T19:33:29.6321796Z ============================== 1 passed in 0.02s =============================== 2022-03-17T19:33:31.4480004Z Running distributed/pipeline/sync/test_stream ... [2022-03-17 19:33:31.447138] 2022-03-17T19:33:31.4481714Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-03-17 19:33:31.447315] 2022-03-17T19:33:32.4341989Z ============================= test session starts ============================== 2022-03-17T19:33:32.4343282Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:32.4362824Z cachedir: .pytest_cache 2022-03-17T19:33:32.4364353Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:32.4365410Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:32.4366173Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:32.4367083Z plugins: hypothesis-4.53.2 2022-03-17T19:33:32.5252531Z collecting ...  2022-03-17T19:33:32.5253563Z collected 19 items  2022-03-17T19:33:32.5256446Z 2022-03-17T19:33:32.5287457Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2022-03-17T19:33:32.9138799Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%] 2022-03-17T19:33:32.9148136Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2022-03-17T19:33:32.9158864Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%] 2022-03-17T19:33:32.9168272Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2022-03-17T19:33:32.9178294Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%] 2022-03-17T19:33:32.9187879Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2022-03-17T19:33:32.9197809Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%] 2022-03-17T19:33:32.9208354Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2022-03-17T19:33:32.9220280Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%] 2022-03-17T19:33:32.9228941Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2022-03-17T19:33:32.9239123Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%] 2022-03-17T19:33:32.9282244Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2022-03-17T19:33:33.2069345Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%] 2022-03-17T19:33:33.2084542Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%] 2022-03-17T19:33:33.4584680Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%] 2022-03-17T19:33:33.4596602Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2022-03-17T19:33:33.5472390Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%] 2022-03-17T19:33:33.5504315Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%] 2022-03-17T19:33:33.5505109Z 2022-03-17T19:33:33.5505829Z ============================== 19 passed in 1.12s ============================== 2022-03-17T19:33:35.7700685Z Running distributed/pipeline/sync/test_transparency ... [2022-03-17 19:33:35.769388] 2022-03-17T19:33:35.7702483Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-03-17 19:33:35.769527] 2022-03-17T19:33:36.7493012Z ============================= test session starts ============================== 2022-03-17T19:33:36.7493600Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:36.7513644Z cachedir: .pytest_cache 2022-03-17T19:33:36.7514360Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:36.7514846Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:36.7515197Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:36.7515608Z plugins: hypothesis-4.53.2 2022-03-17T19:33:36.7617877Z collecting ...  2022-03-17T19:33:36.7618339Z collected 1 item  2022-03-17T19:33:36.7622080Z 2022-03-17T19:33:36.8327333Z distributed/pipeline/sync/test_transparency.py::test_simple_linears libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:33:36.8877602Z PASSED [100%] 2022-03-17T19:33:36.8878048Z 2022-03-17T19:33:36.8878799Z ============================== 1 passed in 0.14s =============================== 2022-03-17T19:33:38.5304960Z Running distributed/pipeline/sync/test_worker ... [2022-03-17 19:33:38.529568] 2022-03-17T19:33:38.5307086Z Executing ['/opt/conda/bin/python3.7', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-03-17 19:33:38.529746] 2022-03-17T19:33:39.4662738Z ============================= test session starts ============================== 2022-03-17T19:33:39.4663342Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3.7 2022-03-17T19:33:39.4686529Z cachedir: .pytest_cache 2022-03-17T19:33:39.4687301Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-03-17T19:33:39.4687798Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:33:39.4688158Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-03-17T19:33:39.4688585Z plugins: hypothesis-4.53.2 2022-03-17T19:33:39.4881237Z collecting ...  2022-03-17T19:33:39.4881703Z collected 6 items  2022-03-17T19:33:39.4882296Z 2022-03-17T19:33:39.4926136Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%] 2022-03-17T19:33:39.4946688Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED [ 33%] 2022-03-17T19:33:39.4967703Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED [ 50%] 2022-03-17T19:33:39.5018976Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED [ 66%] 2022-03-17T19:33:39.5037030Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED [ 83%] 2022-03-17T19:33:39.5067940Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED [100%] 2022-03-17T19:33:39.5068616Z 2022-03-17T19:33:39.5069371Z ============================== 6 passed in 0.04s =============================== 2022-03-17T19:33:41.3460196Z Running distributed/test_c10d_common ... [2022-03-17 19:33:41.345191] 2022-03-17T19:33:41.3462018Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:33:41.345363] 2022-03-17T19:33:42.2043736Z test_debug_level (__main__.CommTest) 2022-03-17T19:33:42.2044179Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:33:42.2044898Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:33:42.2045385Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:33:42.2045849Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:33:42.2046321Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:33:42.2046798Z test_collectives (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:33:42.2047278Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:33:42.2047729Z test_send_recv (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:33:43.0270143Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:33:43.0284207Z 2022-03-17T19:33:43.0284476Z Running tests... 2022-03-17T19:33:43.0284910Z ---------------------------------------------------------------------- 2022-03-17T19:33:43.6107712Z test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17002 2022-03-17T19:33:43.6139753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17003 2022-03-17T19:33:44.5232432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:44.5355571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:46.3207882Z ok (3.292s) 2022-03-17T19:33:46.3208277Z 2022-03-17T19:33:46.3209391Z ---------------------------------------------------------------------- 2022-03-17T19:33:46.3210942Z Ran 1 test in 3.292s 2022-03-17T19:33:46.3211294Z 2022-03-17T19:33:46.3211503Z OK 2022-03-17T19:33:46.3211805Z 2022-03-17T19:33:46.3212098Z Generating XML reports... 2022-03-17T19:33:46.3300810Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220317193343.xml 2022-03-17T19:33:48.7364034Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:33:48.7378990Z 2022-03-17T19:33:48.7379248Z Running tests... 2022-03-17T19:33:48.7380181Z ---------------------------------------------------------------------- 2022-03-17T19:33:49.3158746Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.578s) 2022-03-17T19:33:49.3159081Z 2022-03-17T19:33:49.3159614Z ---------------------------------------------------------------------- 2022-03-17T19:33:49.3159978Z Ran 1 test in 0.578s 2022-03-17T19:33:49.3160150Z 2022-03-17T19:33:49.3160234Z OK 2022-03-17T19:33:49.3160378Z 2022-03-17T19:33:49.3160500Z Generating XML reports... 2022-03-17T19:33:49.3197185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193348.xml 2022-03-17T19:33:51.6713568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:33:51.6723817Z 2022-03-17T19:33:51.6724222Z Running tests... 2022-03-17T19:33:51.6725633Z ---------------------------------------------------------------------- 2022-03-17T19:33:52.2486297Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.576s) 2022-03-17T19:33:52.2487074Z 2022-03-17T19:33:52.2488064Z ---------------------------------------------------------------------- 2022-03-17T19:33:52.2489052Z Ran 1 test in 0.576s 2022-03-17T19:33:52.2489409Z 2022-03-17T19:33:52.2489613Z OK 2022-03-17T19:33:52.2489912Z 2022-03-17T19:33:52.2490188Z Generating XML reports... 2022-03-17T19:33:52.2522963Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193351.xml 2022-03-17T19:33:54.5518862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:33:54.5532542Z 2022-03-17T19:33:54.5532903Z Running tests... 2022-03-17T19:33:54.5533493Z ---------------------------------------------------------------------- 2022-03-17T19:33:55.1309446Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.578s) 2022-03-17T19:33:55.1310236Z 2022-03-17T19:33:55.1310638Z ---------------------------------------------------------------------- 2022-03-17T19:33:55.1310982Z Ran 1 test in 0.578s 2022-03-17T19:33:55.1311153Z 2022-03-17T19:33:55.1311242Z OK 2022-03-17T19:33:55.1311378Z 2022-03-17T19:33:55.1311504Z Generating XML reports... 2022-03-17T19:33:55.1344402Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193354.xml 2022-03-17T19:33:57.9468074Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:33:57.9483580Z 2022-03-17T19:33:57.9483739Z Running tests... 2022-03-17T19:33:57.9484290Z ---------------------------------------------------------------------- 2022-03-17T19:33:58.5265089Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.578s) 2022-03-17T19:33:58.5265756Z 2022-03-17T19:33:58.5266710Z ---------------------------------------------------------------------- 2022-03-17T19:33:58.5267544Z Ran 1 test in 0.578s 2022-03-17T19:33:58.5267911Z 2022-03-17T19:33:58.5268108Z OK 2022-03-17T19:33:58.5268390Z 2022-03-17T19:33:58.5268679Z Generating XML reports... 2022-03-17T19:33:58.5302643Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193357.xml 2022-03-17T19:34:00.9205397Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:34:00.9220022Z 2022-03-17T19:34:00.9220407Z Running tests... 2022-03-17T19:34:00.9221127Z ---------------------------------------------------------------------- 2022-03-17T19:34:01.5139374Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17474 2022-03-17T19:34:01.5191074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17475 2022-03-17T19:34:01.5244901Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17476 2022-03-17T19:34:01.5287208Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17477 2022-03-17T19:34:02.5079484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:02.5251468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:34:02.5265495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:34:02.5272342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:05.0391800Z ok (4.117s) 2022-03-17T19:34:05.0392231Z 2022-03-17T19:34:05.0393298Z ---------------------------------------------------------------------- 2022-03-17T19:34:05.0394267Z Ran 1 test in 4.117s 2022-03-17T19:34:05.0394643Z 2022-03-17T19:34:05.0394859Z OK 2022-03-17T19:34:05.0395139Z 2022-03-17T19:34:05.0395433Z Generating XML reports... 2022-03-17T19:34:05.0477587Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193400.xml 2022-03-17T19:34:07.4116948Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:34:07.4130922Z 2022-03-17T19:34:07.4131323Z Running tests... 2022-03-17T19:34:07.4131781Z ---------------------------------------------------------------------- 2022-03-17T19:34:07.9998803Z test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17814 2022-03-17T19:34:08.0035123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17815 2022-03-17T19:34:08.0079520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17816 2022-03-17T19:34:08.0128556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17817 2022-03-17T19:34:08.9026130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:34:08.9028264Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-03-17T19:34:08.9064799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:08.9299018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:08.9306312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:08.9667718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:34:08.9676365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:34:09.9037939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:34:09.9058444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:09.9060533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:09.9140606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:09.9142530Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:09.9144866Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:13.2258727Z ok (5.812s) 2022-03-17T19:34:13.2259206Z 2022-03-17T19:34:13.2260290Z ---------------------------------------------------------------------- 2022-03-17T19:34:13.2261066Z Ran 1 test in 5.812s 2022-03-17T19:34:13.2261435Z 2022-03-17T19:34:13.2261634Z OK 2022-03-17T19:34:13.2262648Z 2022-03-17T19:34:13.2263208Z Generating XML reports... 2022-03-17T19:34:13.2347269Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193407.xml 2022-03-17T19:34:15.5707277Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:34:15.5721007Z 2022-03-17T19:34:15.5721432Z Running tests... 2022-03-17T19:34:15.5722901Z ---------------------------------------------------------------------- 2022-03-17T19:34:16.1587296Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18159 2022-03-17T19:34:16.1616904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18160 2022-03-17T19:34:16.1660752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18161 2022-03-17T19:34:16.1695412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18162 2022-03-17T19:34:17.1431562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:17.1667681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:17.1732608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:34:17.2262643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:34:19.3863926Z ok (3.814s) 2022-03-17T19:34:19.3864310Z 2022-03-17T19:34:19.3865933Z ---------------------------------------------------------------------- 2022-03-17T19:34:19.3866812Z Ran 1 test in 3.814s 2022-03-17T19:34:19.3867187Z 2022-03-17T19:34:19.3867355Z OK 2022-03-17T19:34:19.3867655Z 2022-03-17T19:34:19.3867944Z Generating XML reports... 2022-03-17T19:34:19.3946204Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193415.xml 2022-03-17T19:34:21.7708391Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:34:21.7723111Z 2022-03-17T19:34:21.7723716Z Running tests... 2022-03-17T19:34:21.7724900Z ---------------------------------------------------------------------- 2022-03-17T19:34:22.3578918Z test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18499 2022-03-17T19:34:22.3611559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18500 2022-03-17T19:34:22.3654757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18501 2022-03-17T19:34:22.3692604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18502 2022-03-17T19:34:23.2659838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:34:23.2661821Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-03-17T19:34:23.3179731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:23.3184049Z [W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-03-17T19:34:23.3326975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:23.3385486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:34:23.3397207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:34:24.2676897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:34:24.3200675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:24.3298097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:24.3299956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:24.3301916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:24.3303692Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:24.3305437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:34:26.2809505Z ok (4.508s) 2022-03-17T19:34:26.2809924Z 2022-03-17T19:34:26.2812092Z ---------------------------------------------------------------------- 2022-03-17T19:34:26.2812930Z Ran 1 test in 4.509s 2022-03-17T19:34:26.2813277Z 2022-03-17T19:34:26.2813471Z OK 2022-03-17T19:34:26.2813777Z 2022-03-17T19:34:26.2814054Z Generating XML reports... 2022-03-17T19:34:26.2893997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193421.xml 2022-03-17T19:34:29.2629658Z Running distributed/test_c10d_gloo ... [2022-03-17 19:34:29.262124] 2022-03-17T19:34:29.2631600Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:34:29.262288] 2022-03-17T19:34:30.1341711Z , <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>]> 2022-03-17T19:34:30.1344910Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) 2022-03-17T19:34:30.1345807Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) 2022-03-17T19:34:30.1346644Z test_gloo_barrier_device_ids (__main__.CommTest) 2022-03-17T19:34:30.1347486Z test_gloo_warn_not_in_group (__main__.CommTest) 2022-03-17T19:34:30.1348321Z test_sequence_num_incremented_gloo_default (__main__.CommTest) 2022-03-17T19:34:30.1349244Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) 2022-03-17T19:34:30.1350151Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) 2022-03-17T19:34:30.1350986Z test_sequence_num_set_gloo_new_group (__main__.CommTest) 2022-03-17T19:34:30.1365588Z , <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>]> 2022-03-17T19:34:30.1380085Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1381281Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1382488Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1383660Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1384931Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1386246Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1387472Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1388638Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1389807Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1391030Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1392277Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1393552Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1394800Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1396092Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1397200Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1398293Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1399378Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1400425Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1401497Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1402806Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1403976Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1405139Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1406334Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1407568Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1408796Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1409909Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1410934Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1411964Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1413010Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1414063Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1415104Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1416189Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1417193Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1418179Z test_sparse_gradients (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1419214Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:34:30.1420120Z 2022-03-17T19:34:30.1433780Z , <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]> 2022-03-17T19:34:30.1447801Z test_allgather_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1448727Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1449625Z test_allgather_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1450555Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1451540Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1452462Z test_allgather_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1453377Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1454272Z test_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1455193Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1455948Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1456414Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1456766Z test_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1457101Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1457460Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1457809Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1458172Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1458544Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1458879Z test_allreduce_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1459212Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1459545Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1459873Z test_broadcast_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1460202Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1460531Z test_broadcast_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1460857Z test_broadcast_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1461181Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1461509Z test_empty_tensors (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1461820Z test_gather_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1462142Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1462513Z test_gather_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1462822Z test_gather_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1463145Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1463479Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1463811Z test_reduce_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1464137Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1464453Z test_reduce_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1464767Z test_reduce_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1465087Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1465407Z test_round_robin (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1465735Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1466066Z test_scatter_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1466393Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1466706Z test_scatter_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1467025Z test_scatter_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1467347Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1467676Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1468017Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1468365Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1468725Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:34:30.1469549Z , <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]> 2022-03-17T19:34:30.1470312Z test_forward_backward (__main__.ReducerTest) 2022-03-17T19:34:30.1470615Z test_forward_backward_optimizer (__main__.ReducerTest) 2022-03-17T19:34:30.1470943Z test_forward_backward_unused_parameters (__main__.ReducerTest) 2022-03-17T19:34:30.1471266Z test_multi_dtype_multi_bucket (__main__.ReducerTest) 2022-03-17T19:34:30.1471615Z test_multi_dtype_single_bucket (__main__.ReducerTest) 2022-03-17T19:34:30.1471930Z test_single_dtype_single_bucket (__main__.ReducerTest) 2022-03-17T19:34:30.1472306Z ]> 2022-03-17T19:34:30.1472664Z test_logging_init (__main__.RendezvousEnvTest) 2022-03-17T19:34:30.1472948Z 2022-03-17T19:34:30.1473315Z ]> 2022-03-17T19:34:30.1473714Z test_default_store_timeout_gloo (__main__.TimeoutTest) 2022-03-17T19:34:31.0004311Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:34:31.0020888Z 2022-03-17T19:34:31.0021234Z Running tests... 2022-03-17T19:34:31.0021793Z ---------------------------------------------------------------------- 2022-03-17T19:34:31.5891412Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18911 2022-03-17T19:34:31.5934586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18912 2022-03-17T19:34:32.4814789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:32.5164018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:34.4016364Z ok (3.399s) 2022-03-17T19:34:34.4016728Z 2022-03-17T19:34:34.4017625Z ---------------------------------------------------------------------- 2022-03-17T19:34:34.4019205Z Ran 1 test in 3.400s 2022-03-17T19:34:34.4019580Z 2022-03-17T19:34:34.4019749Z OK 2022-03-17T19:34:34.4020044Z 2022-03-17T19:34:34.4020322Z Generating XML reports... 2022-03-17T19:34:34.4098149Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193430.xml 2022-03-17T19:34:36.9170879Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:34:36.9184147Z 2022-03-17T19:34:36.9184452Z Running tests... 2022-03-17T19:34:36.9184993Z ---------------------------------------------------------------------- 2022-03-17T19:34:37.5035190Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19121 2022-03-17T19:34:37.5087711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19122 2022-03-17T19:34:38.3683239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:38.4126322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:41.4166949Z ok (4.498s) 2022-03-17T19:34:41.4167339Z 2022-03-17T19:34:41.4168758Z ---------------------------------------------------------------------- 2022-03-17T19:34:41.4169565Z Ran 1 test in 4.498s 2022-03-17T19:34:41.4169934Z 2022-03-17T19:34:41.4170127Z OK 2022-03-17T19:34:41.4170429Z 2022-03-17T19:34:41.4170742Z Generating XML reports... 2022-03-17T19:34:41.4249472Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193436.xml 2022-03-17T19:34:43.8232483Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:34:43.8246814Z 2022-03-17T19:34:43.8247218Z Running tests... 2022-03-17T19:34:43.8247873Z ---------------------------------------------------------------------- 2022-03-17T19:34:44.4114095Z test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19331 2022-03-17T19:34:44.4147717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19332 2022-03-17T19:34:45.2745491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:45.3386657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:45.3557979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:45.3559943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:45.3561241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:45.3563096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:47.2219089Z ok (3.397s) 2022-03-17T19:34:47.2219980Z 2022-03-17T19:34:47.2221312Z ---------------------------------------------------------------------- 2022-03-17T19:34:47.2222116Z Ran 1 test in 3.397s 2022-03-17T19:34:47.2222483Z 2022-03-17T19:34:47.2222695Z OK 2022-03-17T19:34:47.2222989Z 2022-03-17T19:34:47.2223273Z Generating XML reports... 2022-03-17T19:34:47.2299875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193443.xml 2022-03-17T19:34:49.6921273Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:34:49.6934111Z 2022-03-17T19:34:49.6934574Z Running tests... 2022-03-17T19:34:49.6935757Z ---------------------------------------------------------------------- 2022-03-17T19:34:50.2803991Z test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19541 2022-03-17T19:34:50.2840086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19542 2022-03-17T19:34:51.1480840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:51.2149207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:51.2298718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:51.2300073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:51.2302163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:51.2304000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:51.2312578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:34:51.2406838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:34:51.2408584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:34:51.2416481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:34:53.9946319Z ok (4.301s) 2022-03-17T19:34:53.9946711Z 2022-03-17T19:34:53.9947803Z ---------------------------------------------------------------------- 2022-03-17T19:34:53.9948825Z Ran 1 test in 4.301s 2022-03-17T19:34:53.9949200Z 2022-03-17T19:34:53.9949411Z OK 2022-03-17T19:34:53.9949709Z 2022-03-17T19:34:53.9949998Z Generating XML reports... 2022-03-17T19:34:54.0024210Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193449.xml 2022-03-17T19:34:56.4158035Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:34:56.4171174Z 2022-03-17T19:34:56.4171448Z Running tests... 2022-03-17T19:34:56.4171890Z ---------------------------------------------------------------------- 2022-03-17T19:34:57.0020441Z test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19754 2022-03-17T19:34:57.0062145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19755 2022-03-17T19:34:57.9287312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:57.9357693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:57.9497881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:57.9499975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:57.9501932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:57.9603902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:57.9807122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:34:57.9808532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:34:57.9809960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:34:57.9910746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:35:00.7169221Z ok (4.299s) 2022-03-17T19:35:00.7169626Z 2022-03-17T19:35:00.7170524Z ---------------------------------------------------------------------- 2022-03-17T19:35:00.7171303Z Ran 1 test in 4.299s 2022-03-17T19:35:00.7172244Z 2022-03-17T19:35:00.7172438Z OK 2022-03-17T19:35:00.7172736Z 2022-03-17T19:35:00.7173016Z Generating XML reports... 2022-03-17T19:35:00.7249345Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193456.xml 2022-03-17T19:35:03.4418909Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:35:03.4431868Z 2022-03-17T19:35:03.4432552Z Running tests... 2022-03-17T19:35:03.4434013Z ---------------------------------------------------------------------- 2022-03-17T19:35:04.0285283Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19970 2022-03-17T19:35:04.0319507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19971 2022-03-17T19:35:04.8987811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:04.9042556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:06.5394182Z skip (3.096s) 2022-03-17T19:35:06.5394600Z 2022-03-17T19:35:06.5395666Z ---------------------------------------------------------------------- 2022-03-17T19:35:06.5396419Z Ran 1 test in 3.096s 2022-03-17T19:35:06.5396792Z 2022-03-17T19:35:06.5397037Z OK (skipped=1) 2022-03-17T19:35:06.5397383Z 2022-03-17T19:35:06.5397653Z Generating XML reports... 2022-03-17T19:35:06.5475926Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193503.xml 2022-03-17T19:35:08.9646640Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:35:08.9658506Z 2022-03-17T19:35:08.9658638Z Running tests... 2022-03-17T19:35:08.9661751Z ---------------------------------------------------------------------- 2022-03-17T19:35:09.5500828Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20174 2022-03-17T19:35:09.5532158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20175 2022-03-17T19:35:10.4775564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:10.4855952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:10.5069303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:35:10.5071188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:35:10.5072500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:35:10.5074148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:35:12.3605891Z ok (3.394s) 2022-03-17T19:35:12.3606322Z 2022-03-17T19:35:12.3607618Z ---------------------------------------------------------------------- 2022-03-17T19:35:12.3608434Z Ran 1 test in 3.395s 2022-03-17T19:35:12.3608796Z 2022-03-17T19:35:12.3608995Z OK 2022-03-17T19:35:12.3609308Z 2022-03-17T19:35:12.3609572Z Generating XML reports... 2022-03-17T19:35:12.3687066Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193508.xml 2022-03-17T19:35:14.7004217Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:35:14.7015040Z 2022-03-17T19:35:14.7015947Z Running tests... 2022-03-17T19:35:14.7017388Z ---------------------------------------------------------------------- 2022-03-17T19:35:15.2877364Z test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20384 2022-03-17T19:35:15.2911452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20385 2022-03-17T19:35:16.1615901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:16.1936133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:16.2046294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:35:16.2047408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:35:16.2049216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:35:16.2050986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:35:16.2356538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:35:16.2357661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:35:16.2359461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:35:16.2459652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:35:18.0995854Z ok (3.398s) 2022-03-17T19:35:18.0996420Z 2022-03-17T19:35:18.0997662Z ---------------------------------------------------------------------- 2022-03-17T19:35:18.0998736Z Ran 1 test in 3.398s 2022-03-17T19:35:18.0999221Z 2022-03-17T19:35:18.0999465Z OK 2022-03-17T19:35:18.0999824Z 2022-03-17T19:35:18.1000082Z Generating XML reports... 2022-03-17T19:35:18.1079626Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193514.xml 2022-03-17T19:35:20.8287259Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:35:20.8298511Z 2022-03-17T19:35:20.8298766Z Running tests... 2022-03-17T19:35:20.8299219Z ---------------------------------------------------------------------- 2022-03-17T19:35:20.8306925Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:35:21.4155252Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20600 2022-03-17T19:35:21.4191582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20601 2022-03-17T19:35:22.2791722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:22.3662975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:35.0495206Z ok (14.219s) 2022-03-17T19:35:35.0495673Z 2022-03-17T19:35:35.0496637Z ---------------------------------------------------------------------- 2022-03-17T19:35:35.0497447Z Ran 1 test in 14.219s 2022-03-17T19:35:35.0497796Z 2022-03-17T19:35:35.0498015Z OK 2022-03-17T19:35:35.0498319Z 2022-03-17T19:35:35.0498603Z Generating XML reports... 2022-03-17T19:35:35.0577403Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193520.xml 2022-03-17T19:35:37.5299100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:35:37.5312064Z 2022-03-17T19:35:37.5312488Z Running tests... 2022-03-17T19:35:37.5313499Z ---------------------------------------------------------------------- 2022-03-17T19:35:37.5319589Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:35:38.1365869Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20814 2022-03-17T19:35:38.1411768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20815 2022-03-17T19:35:39.0589913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:39.0725666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:51.7646428Z ok (14.233s) 2022-03-17T19:35:51.7646770Z 2022-03-17T19:35:51.7647700Z ---------------------------------------------------------------------- 2022-03-17T19:35:51.7648485Z Ran 1 test in 14.233s 2022-03-17T19:35:51.7648847Z 2022-03-17T19:35:51.7649059Z OK 2022-03-17T19:35:51.7649340Z 2022-03-17T19:35:51.7649666Z Generating XML reports... 2022-03-17T19:35:51.7771631Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193537.xml 2022-03-17T19:35:54.2125915Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:35:54.2137742Z 2022-03-17T19:35:54.2138007Z Running tests... 2022-03-17T19:35:54.2138459Z ---------------------------------------------------------------------- 2022-03-17T19:35:54.2148024Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:35:54.8008327Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21028 2022-03-17T19:35:54.8050851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21029 2022-03-17T19:35:55.7132116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:55.7330098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:05.4495714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.4497014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.5002697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.5004025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.5239666Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:36:05.5242709Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:36:05.5246091Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:36:05.5248362Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:36:05.5423680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.5424960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.5764093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.5765499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.6229697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.6230932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.6639953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:05.6641177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:08.4365683Z ok (14.222s) 2022-03-17T19:36:08.4366117Z 2022-03-17T19:36:08.4367176Z ---------------------------------------------------------------------- 2022-03-17T19:36:08.4367965Z Ran 1 test in 14.222s 2022-03-17T19:36:08.4371051Z 2022-03-17T19:36:08.4372182Z OK 2022-03-17T19:36:08.4372644Z 2022-03-17T19:36:08.4372948Z Generating XML reports... 2022-03-17T19:36:08.4445040Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193554.xml 2022-03-17T19:36:10.8064324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:36:10.8074926Z 2022-03-17T19:36:10.8075205Z Running tests... 2022-03-17T19:36:10.8075779Z ---------------------------------------------------------------------- 2022-03-17T19:36:10.8085929Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:36:11.3962854Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21242 2022-03-17T19:36:11.3999979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21243 2022-03-17T19:36:12.3157348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:12.3374709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:22.0418081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.0419944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.0965723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.0966950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.1217742Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:36:22.1219961Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:36:22.1223603Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:36:22.1225728Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:36:22.1415989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.1417237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.1775645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.1776852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.2272791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.2274196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.2685118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:22.2686380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:25.0200765Z ok (14.212s) 2022-03-17T19:36:25.0201147Z 2022-03-17T19:36:25.0202254Z ---------------------------------------------------------------------- 2022-03-17T19:36:25.0203013Z Ran 1 test in 14.213s 2022-03-17T19:36:25.0203815Z 2022-03-17T19:36:25.0204000Z OK 2022-03-17T19:36:25.0204288Z 2022-03-17T19:36:25.0204568Z Generating XML reports... 2022-03-17T19:36:25.0284353Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193610.xml 2022-03-17T19:36:27.4277239Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:36:27.4288863Z 2022-03-17T19:36:27.4289514Z Running tests... 2022-03-17T19:36:27.4290733Z ---------------------------------------------------------------------- 2022-03-17T19:36:27.4296112Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:36:28.0133005Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21456 2022-03-17T19:36:28.0176459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21457 2022-03-17T19:36:28.9142226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:28.9297103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:38.7375026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:38.7376394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:38.7852146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:38.7853394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:41.4495892Z ok (14.020s) 2022-03-17T19:36:41.4496456Z 2022-03-17T19:36:41.4497728Z ---------------------------------------------------------------------- 2022-03-17T19:36:41.4501926Z Ran 1 test in 14.021s 2022-03-17T19:36:41.4502365Z 2022-03-17T19:36:41.4502563Z OK 2022-03-17T19:36:41.4502889Z 2022-03-17T19:36:41.4503183Z Generating XML reports... 2022-03-17T19:36:41.4576820Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193627.xml 2022-03-17T19:36:43.8036776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:36:43.8049213Z 2022-03-17T19:36:43.8049503Z Running tests... 2022-03-17T19:36:43.8050394Z ---------------------------------------------------------------------- 2022-03-17T19:36:43.8056506Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:36:44.3928525Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21670 2022-03-17T19:36:44.3961639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21671 2022-03-17T19:36:45.3097629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:45.3155235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:55.2205825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:55.2212831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:55.2755506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:55.2756673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:36:57.9229833Z ok (14.118s) 2022-03-17T19:36:57.9230342Z 2022-03-17T19:36:57.9231531Z ---------------------------------------------------------------------- 2022-03-17T19:36:57.9232492Z Ran 1 test in 14.118s 2022-03-17T19:36:57.9232871Z 2022-03-17T19:36:57.9233065Z OK 2022-03-17T19:36:57.9233349Z 2022-03-17T19:36:57.9233648Z Generating XML reports... 2022-03-17T19:36:57.9311764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193643.xml 2022-03-17T19:37:00.2868783Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:37:00.2877212Z 2022-03-17T19:37:00.2877406Z Running tests... 2022-03-17T19:37:00.2878023Z ---------------------------------------------------------------------- 2022-03-17T19:37:00.2892894Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:37:00.8757093Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21884 2022-03-17T19:37:00.8792142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21885 2022-03-17T19:37:01.7892350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:01.8012978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:11.4578179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:11.4579973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:11.4994862Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:37:11.4999540Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:37:11.5599938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:11.5602323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:14.4088987Z ok (14.121s) 2022-03-17T19:37:14.4089456Z 2022-03-17T19:37:14.4090709Z ---------------------------------------------------------------------- 2022-03-17T19:37:14.4091656Z Ran 1 test in 14.121s 2022-03-17T19:37:14.4092028Z 2022-03-17T19:37:14.4092218Z OK 2022-03-17T19:37:14.4095463Z 2022-03-17T19:37:14.4095895Z Generating XML reports... 2022-03-17T19:37:14.4172298Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193700.xml 2022-03-17T19:37:17.0795525Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:37:17.0807261Z 2022-03-17T19:37:17.0807645Z Running tests... 2022-03-17T19:37:17.0808797Z ---------------------------------------------------------------------- 2022-03-17T19:37:17.0819807Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:37:17.6695584Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22098 2022-03-17T19:37:17.6740208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22099 2022-03-17T19:37:18.5871856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:18.5995185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:30.9048311Z ok (13.824s) 2022-03-17T19:37:30.9048933Z 2022-03-17T19:37:30.9050034Z ---------------------------------------------------------------------- 2022-03-17T19:37:30.9050873Z Ran 1 test in 13.824s 2022-03-17T19:37:30.9051237Z 2022-03-17T19:37:30.9051447Z OK 2022-03-17T19:37:30.9051723Z 2022-03-17T19:37:30.9052010Z Generating XML reports... 2022-03-17T19:37:30.9131412Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193717.xml 2022-03-17T19:37:33.3470365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:37:33.3486353Z 2022-03-17T19:37:33.3486537Z Running tests... 2022-03-17T19:37:33.3487208Z ---------------------------------------------------------------------- 2022-03-17T19:37:33.3492613Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:37:33.9429422Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22312 2022-03-17T19:37:33.9472176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22313 2022-03-17T19:37:34.8688380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:34.8904203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:44.5946641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:44.5948063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:44.6437094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:44.6438404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:37:47.2738174Z ok (13.925s) 2022-03-17T19:37:47.2738691Z 2022-03-17T19:37:47.2739918Z ---------------------------------------------------------------------- 2022-03-17T19:37:47.2740787Z Ran 1 test in 13.925s 2022-03-17T19:37:47.2741158Z 2022-03-17T19:37:47.2741347Z OK 2022-03-17T19:37:47.2741643Z 2022-03-17T19:37:47.2741933Z Generating XML reports... 2022-03-17T19:37:47.2818813Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193733.xml 2022-03-17T19:37:49.6825635Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:37:49.6837789Z 2022-03-17T19:37:49.6837917Z Running tests... 2022-03-17T19:37:49.6838520Z ---------------------------------------------------------------------- 2022-03-17T19:37:49.6850628Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:37:50.2754498Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22526 2022-03-17T19:37:50.2796004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22527 2022-03-17T19:37:51.1748484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:51.1995779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:00.6599260Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:38:00.7343218Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:38:00.9183972Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:38:00.9186174Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:38:00.9189346Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:38:00.9191449Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:38:00.9365191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:00.9369375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:01.0181669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:01.0182897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:03.8101163Z ok (14.126s) 2022-03-17T19:38:03.8101658Z 2022-03-17T19:38:03.8102812Z ---------------------------------------------------------------------- 2022-03-17T19:38:03.8103596Z Ran 1 test in 14.126s 2022-03-17T19:38:03.8103980Z 2022-03-17T19:38:03.8104167Z OK 2022-03-17T19:38:03.8104468Z 2022-03-17T19:38:03.8104754Z Generating XML reports... 2022-03-17T19:38:03.8157773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193749.xml 2022-03-17T19:38:06.1767704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:38:06.1778937Z 2022-03-17T19:38:06.1779070Z Running tests... 2022-03-17T19:38:06.1779724Z ---------------------------------------------------------------------- 2022-03-17T19:38:06.1791761Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:38:06.7658250Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22740 2022-03-17T19:38:06.7693284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22741 2022-03-17T19:38:07.6851084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:07.6909641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:17.5024431Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:38:17.5026875Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:38:17.5030049Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:38:17.5032757Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:38:17.6095264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:17.6097800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:17.6742557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:17.6743872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:20.3971248Z ok (14.218s) 2022-03-17T19:38:20.3971739Z 2022-03-17T19:38:20.3972943Z ---------------------------------------------------------------------- 2022-03-17T19:38:20.3973779Z Ran 1 test in 14.219s 2022-03-17T19:38:20.3974154Z 2022-03-17T19:38:20.3974333Z OK 2022-03-17T19:38:20.3974632Z 2022-03-17T19:38:20.3974919Z Generating XML reports... 2022-03-17T19:38:20.4051875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193806.xml 2022-03-17T19:38:22.8083221Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:38:22.8095408Z 2022-03-17T19:38:22.8095846Z Running tests... 2022-03-17T19:38:22.8096388Z ---------------------------------------------------------------------- 2022-03-17T19:38:22.8111470Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:38:23.3987889Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22954 2022-03-17T19:38:23.4023406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22955 2022-03-17T19:38:24.2850481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:24.3258731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:34.0648774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:34.0650064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:34.1299880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:34.1301249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:36.7244402Z ok (13.914s) 2022-03-17T19:38:36.7244919Z 2022-03-17T19:38:36.7246280Z ---------------------------------------------------------------------- 2022-03-17T19:38:36.7247071Z Ran 1 test in 13.915s 2022-03-17T19:38:36.7247439Z 2022-03-17T19:38:36.7247688Z OK 2022-03-17T19:38:36.7248011Z 2022-03-17T19:38:36.7248282Z Generating XML reports... 2022-03-17T19:38:36.7326002Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193822.xml 2022-03-17T19:38:39.1634036Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:38:39.1648315Z 2022-03-17T19:38:39.1648735Z Running tests... 2022-03-17T19:38:39.1649364Z ---------------------------------------------------------------------- 2022-03-17T19:38:39.1667744Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:38:39.7708507Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23168 2022-03-17T19:38:39.7753998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23169 2022-03-17T19:38:40.6788364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:40.6980169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:50.6357885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.6367846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.6862584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.6863994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.7192177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.7193422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.7661150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:50.7662486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:38:53.5061398Z ok (14.341s) 2022-03-17T19:38:53.5061955Z 2022-03-17T19:38:53.5063342Z ---------------------------------------------------------------------- 2022-03-17T19:38:53.5064313Z Ran 1 test in 14.341s 2022-03-17T19:38:53.5064659Z 2022-03-17T19:38:53.5064847Z OK 2022-03-17T19:38:53.5065141Z 2022-03-17T19:38:53.5065450Z Generating XML reports... 2022-03-17T19:38:53.5143246Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193839.xml 2022-03-17T19:38:55.9030259Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:38:55.9041530Z 2022-03-17T19:38:55.9042042Z Running tests... 2022-03-17T19:38:55.9042927Z ---------------------------------------------------------------------- 2022-03-17T19:38:55.9050733Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-03-17T19:38:56.4927502Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23382 2022-03-17T19:38:56.4965247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23383 2022-03-17T19:38:57.3575511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:57.3632057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:59.2060523Z ok (3.301s) 2022-03-17T19:38:59.2061074Z 2022-03-17T19:38:59.2062081Z ---------------------------------------------------------------------- 2022-03-17T19:38:59.2062994Z Ran 1 test in 3.302s 2022-03-17T19:38:59.2063527Z 2022-03-17T19:38:59.2064203Z OK 2022-03-17T19:38:59.2064632Z 2022-03-17T19:38:59.2064957Z Generating XML reports... 2022-03-17T19:38:59.2143913Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193855.xml 2022-03-17T19:39:01.6834002Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:01.6847360Z 2022-03-17T19:39:01.6847684Z Running tests... 2022-03-17T19:39:01.6848225Z ---------------------------------------------------------------------- 2022-03-17T19:39:01.6853542Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-03-17T19:39:02.2757359Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23596 2022-03-17T19:39:02.2792346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23597 2022-03-17T19:39:03.1445897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:03.1905589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:05.7882676Z ok (4.103s) 2022-03-17T19:39:05.7883901Z 2022-03-17T19:39:05.7885033Z ---------------------------------------------------------------------- 2022-03-17T19:39:05.7885946Z Ran 1 test in 4.103s 2022-03-17T19:39:05.7886370Z 2022-03-17T19:39:05.7886637Z OK 2022-03-17T19:39:05.7886980Z 2022-03-17T19:39:05.7887342Z Generating XML reports... 2022-03-17T19:39:05.7965461Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193901.xml 2022-03-17T19:39:08.2104857Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:08.2117603Z 2022-03-17T19:39:08.2117844Z Running tests... 2022-03-17T19:39:08.2118333Z ---------------------------------------------------------------------- 2022-03-17T19:39:08.2131415Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-03-17T19:39:08.8011003Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23810 2022-03-17T19:39:08.8047437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23811 2022-03-17T19:39:09.6610968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:09.7022460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:11.6134259Z ok (3.401s) 2022-03-17T19:39:11.6134642Z 2022-03-17T19:39:11.6135491Z ---------------------------------------------------------------------- 2022-03-17T19:39:11.6136265Z Ran 1 test in 3.402s 2022-03-17T19:39:11.6136627Z 2022-03-17T19:39:11.6136829Z OK 2022-03-17T19:39:11.6137105Z 2022-03-17T19:39:11.6137378Z Generating XML reports... 2022-03-17T19:39:11.6217063Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193908.xml 2022-03-17T19:39:14.2766037Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:14.2783897Z 2022-03-17T19:39:14.2784248Z Running tests... 2022-03-17T19:39:14.2784768Z ---------------------------------------------------------------------- 2022-03-17T19:39:14.2796059Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-03-17T19:39:14.8647379Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24020 2022-03-17T19:39:14.8687027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24021 2022-03-17T19:39:15.7180583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:15.7862612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:17.7766987Z ok (3.498s) 2022-03-17T19:39:17.7767393Z 2022-03-17T19:39:17.7768351Z ---------------------------------------------------------------------- 2022-03-17T19:39:17.7769141Z Ran 1 test in 3.499s 2022-03-17T19:39:17.7769506Z 2022-03-17T19:39:17.7769706Z OK 2022-03-17T19:39:17.7770638Z 2022-03-17T19:39:17.7770917Z Generating XML reports... 2022-03-17T19:39:17.7851155Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193914.xml 2022-03-17T19:39:20.2460837Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:20.2478073Z 2022-03-17T19:39:20.2478517Z Running tests... 2022-03-17T19:39:20.2479060Z ---------------------------------------------------------------------- 2022-03-17T19:39:20.2485355Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-03-17T19:39:20.8357370Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24296 2022-03-17T19:39:20.8388697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24297 2022-03-17T19:39:21.7295481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:21.7494182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:23.6506430Z ok (3.403s) 2022-03-17T19:39:23.6506995Z 2022-03-17T19:39:23.6508062Z ---------------------------------------------------------------------- 2022-03-17T19:39:23.6509021Z Ran 1 test in 3.403s 2022-03-17T19:39:23.6509473Z 2022-03-17T19:39:23.6509709Z OK 2022-03-17T19:39:23.6510031Z 2022-03-17T19:39:23.6510322Z Generating XML reports... 2022-03-17T19:39:23.6582709Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193920.xml 2022-03-17T19:39:26.3142249Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:26.3154395Z 2022-03-17T19:39:26.3154831Z Running tests... 2022-03-17T19:39:26.3155295Z ---------------------------------------------------------------------- 2022-03-17T19:39:26.3172165Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-03-17T19:39:26.9109527Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24506 2022-03-17T19:39:26.9146568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24507 2022-03-17T19:39:27.7695947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:27.8676714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:29.7224155Z ok (3.406s) 2022-03-17T19:39:29.7224589Z 2022-03-17T19:39:29.7225809Z ---------------------------------------------------------------------- 2022-03-17T19:39:29.7226741Z Ran 1 test in 3.407s 2022-03-17T19:39:29.7227180Z 2022-03-17T19:39:29.7227394Z OK 2022-03-17T19:39:29.7227723Z 2022-03-17T19:39:29.7228072Z Generating XML reports... 2022-03-17T19:39:29.7302920Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193926.xml 2022-03-17T19:39:32.1256401Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:32.1265197Z 2022-03-17T19:39:32.1265457Z Running tests... 2022-03-17T19:39:32.1266555Z ---------------------------------------------------------------------- 2022-03-17T19:39:32.1290846Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-03-17T19:39:32.7166381Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24720 2022-03-17T19:39:32.7199406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24721 2022-03-17T19:39:33.6384422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:33.6405609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:33.6650951Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:39:36.7299760Z ok (4.603s) 2022-03-17T19:39:36.7300132Z 2022-03-17T19:39:36.7301054Z ---------------------------------------------------------------------- 2022-03-17T19:39:36.7301820Z Ran 1 test in 4.603s 2022-03-17T19:39:36.7302178Z 2022-03-17T19:39:36.7302346Z OK 2022-03-17T19:39:36.7302634Z 2022-03-17T19:39:36.7302905Z Generating XML reports... 2022-03-17T19:39:36.7378215Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193932.xml 2022-03-17T19:39:39.1041628Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:39.1054131Z 2022-03-17T19:39:39.1054793Z Running tests... 2022-03-17T19:39:39.1055939Z ---------------------------------------------------------------------- 2022-03-17T19:39:39.6917515Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24934 2022-03-17T19:39:39.6986818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24935 2022-03-17T19:39:40.6068489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:40.6238137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:42.8095339Z ok (3.704s) 2022-03-17T19:39:42.8095755Z 2022-03-17T19:39:42.8098821Z ---------------------------------------------------------------------- 2022-03-17T19:39:42.8100009Z Ran 1 test in 3.704s 2022-03-17T19:39:42.8100459Z 2022-03-17T19:39:42.8100647Z OK 2022-03-17T19:39:42.8100944Z 2022-03-17T19:39:42.8101216Z Generating XML reports... 2022-03-17T19:39:42.8181047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193939.xml 2022-03-17T19:39:45.3349997Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:45.3360178Z 2022-03-17T19:39:45.3360405Z Running tests... 2022-03-17T19:39:45.3360840Z ---------------------------------------------------------------------- 2022-03-17T19:39:45.9234538Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25148 2022-03-17T19:39:45.9270986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25149 2022-03-17T19:39:46.8195468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:46.8573699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:49.3355800Z ok (3.999s) 2022-03-17T19:39:49.3356207Z 2022-03-17T19:39:49.3357165Z ---------------------------------------------------------------------- 2022-03-17T19:39:49.3358666Z Ran 1 test in 3.999s 2022-03-17T19:39:49.3359053Z 2022-03-17T19:39:49.3359239Z OK 2022-03-17T19:39:49.3359536Z 2022-03-17T19:39:49.3359799Z Generating XML reports... 2022-03-17T19:39:49.3437261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193945.xml 2022-03-17T19:39:51.7377485Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:51.7390005Z 2022-03-17T19:39:51.7390286Z Running tests... 2022-03-17T19:39:51.7390734Z ---------------------------------------------------------------------- 2022-03-17T19:39:52.3282289Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25362 2022-03-17T19:39:52.3317587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25363 2022-03-17T19:39:53.2405077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:53.2503726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:53.2698160Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:39:53.2700801Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:39:53.2703932Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:39:53.2706014Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:39:55.5387422Z ok (3.799s) 2022-03-17T19:39:55.5387850Z 2022-03-17T19:39:55.5388835Z ---------------------------------------------------------------------- 2022-03-17T19:39:55.5389582Z Ran 1 test in 3.799s 2022-03-17T19:39:55.5389944Z 2022-03-17T19:39:55.5392773Z OK 2022-03-17T19:39:55.5393163Z 2022-03-17T19:39:55.5393512Z Generating XML reports... 2022-03-17T19:39:55.5468049Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193951.xml 2022-03-17T19:39:57.9343776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:39:57.9355767Z 2022-03-17T19:39:57.9356065Z Running tests... 2022-03-17T19:39:57.9356559Z ---------------------------------------------------------------------- 2022-03-17T19:39:58.5240681Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25576 2022-03-17T19:39:58.5279562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25577 2022-03-17T19:39:59.4381544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:59.4424059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:09.1596987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:09.1598530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:11.8588979Z ok (13.923s) 2022-03-17T19:40:11.8589595Z 2022-03-17T19:40:11.8590752Z ---------------------------------------------------------------------- 2022-03-17T19:40:11.8591716Z Ran 1 test in 13.923s 2022-03-17T19:40:11.8592170Z 2022-03-17T19:40:11.8592860Z OK 2022-03-17T19:40:11.8593168Z 2022-03-17T19:40:11.8593451Z Generating XML reports... 2022-03-17T19:40:11.8668176Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193957.xml 2022-03-17T19:40:14.2282824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:40:14.2293802Z 2022-03-17T19:40:14.2293975Z Running tests... 2022-03-17T19:40:14.2294790Z ---------------------------------------------------------------------- 2022-03-17T19:40:14.8185181Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25790 2022-03-17T19:40:14.8225273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25791 2022-03-17T19:40:15.6914059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:15.6934858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:25.7564849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:25.7566498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:28.6504271Z ok (14.420s) 2022-03-17T19:40:28.6504779Z 2022-03-17T19:40:28.6506991Z ---------------------------------------------------------------------- 2022-03-17T19:40:28.6508719Z Ran 1 test in 14.421s 2022-03-17T19:40:28.6509127Z 2022-03-17T19:40:28.6509361Z OK 2022-03-17T19:40:28.6509708Z 2022-03-17T19:40:28.6510028Z Generating XML reports... 2022-03-17T19:40:28.6585436Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194014.xml 2022-03-17T19:40:31.0555163Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:40:31.0567884Z 2022-03-17T19:40:31.0568572Z Running tests... 2022-03-17T19:40:31.0569887Z ---------------------------------------------------------------------- 2022-03-17T19:40:31.6436185Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26004 2022-03-17T19:40:31.6471412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26005 2022-03-17T19:40:32.5607506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:32.5669822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:34.5545077Z skip (3.497s) 2022-03-17T19:40:34.5545473Z 2022-03-17T19:40:34.5546948Z ---------------------------------------------------------------------- 2022-03-17T19:40:34.5547963Z Ran 1 test in 3.498s 2022-03-17T19:40:34.5548421Z 2022-03-17T19:40:34.5548734Z OK (skipped=1) 2022-03-17T19:40:34.5549304Z 2022-03-17T19:40:34.5549610Z Generating XML reports... 2022-03-17T19:40:34.5629007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194031.xml 2022-03-17T19:40:36.9801261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:40:36.9814537Z 2022-03-17T19:40:36.9814929Z Running tests... 2022-03-17T19:40:36.9816282Z ---------------------------------------------------------------------- 2022-03-17T19:40:37.5710406Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26208 2022-03-17T19:40:37.5747823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26209 2022-03-17T19:40:38.4389547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:38.5024101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:40.3821724Z skip (3.400s) 2022-03-17T19:40:40.3822374Z 2022-03-17T19:40:40.3823404Z ---------------------------------------------------------------------- 2022-03-17T19:40:40.3824185Z Ran 1 test in 3.400s 2022-03-17T19:40:40.3824559Z 2022-03-17T19:40:40.3824789Z OK (skipped=1) 2022-03-17T19:40:40.3825118Z 2022-03-17T19:40:40.3825386Z Generating XML reports... 2022-03-17T19:40:40.3898809Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194036.xml 2022-03-17T19:40:42.7909096Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:40:42.7921185Z 2022-03-17T19:40:42.7921292Z Running tests... 2022-03-17T19:40:42.7922053Z ---------------------------------------------------------------------- 2022-03-17T19:40:43.3792203Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26412 2022-03-17T19:40:43.3829766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26413 2022-03-17T19:40:44.2451734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:44.3008737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:44.3468207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:44.3469458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:46.8914164Z ok (4.099s) 2022-03-17T19:40:46.8914553Z 2022-03-17T19:40:46.8915931Z ---------------------------------------------------------------------- 2022-03-17T19:40:46.8917086Z Ran 1 test in 4.099s 2022-03-17T19:40:46.8917464Z 2022-03-17T19:40:46.8917658Z OK 2022-03-17T19:40:46.8917975Z 2022-03-17T19:40:46.8918265Z Generating XML reports... 2022-03-17T19:40:46.8996541Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194042.xml 2022-03-17T19:40:49.3217360Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:40:49.3227805Z 2022-03-17T19:40:49.3228490Z Running tests... 2022-03-17T19:40:49.3229815Z ---------------------------------------------------------------------- 2022-03-17T19:40:49.9113201Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26688 2022-03-17T19:40:49.9147298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26689 2022-03-17T19:40:50.8402881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:50.8446764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:50.8923126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:50.8924504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:52.7220899Z ok (3.399s) 2022-03-17T19:40:52.7221099Z 2022-03-17T19:40:52.7221670Z ---------------------------------------------------------------------- 2022-03-17T19:40:52.7222154Z Ran 1 test in 3.399s 2022-03-17T19:40:52.7222387Z 2022-03-17T19:40:52.7222548Z OK 2022-03-17T19:40:52.7222684Z 2022-03-17T19:40:52.7222810Z Generating XML reports... 2022-03-17T19:40:52.7300345Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194049.xml 2022-03-17T19:40:55.1373094Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:40:55.1385555Z 2022-03-17T19:40:55.1386132Z Running tests... 2022-03-17T19:40:55.1387286Z ---------------------------------------------------------------------- 2022-03-17T19:40:55.1406692Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-03-17T19:40:55.7263438Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26964 2022-03-17T19:40:55.7302763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26965 2022-03-17T19:40:56.6125922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:56.6827772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:56.7103440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:56.7104052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:40:58.5369425Z ok (3.398s) 2022-03-17T19:40:58.5369840Z 2022-03-17T19:40:58.5370848Z ---------------------------------------------------------------------- 2022-03-17T19:40:58.5371896Z Ran 1 test in 3.399s 2022-03-17T19:40:58.5372263Z 2022-03-17T19:40:58.5372493Z OK 2022-03-17T19:40:58.5372792Z 2022-03-17T19:40:58.5373080Z Generating XML reports... 2022-03-17T19:40:58.5452839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194055.xml 2022-03-17T19:41:00.9281968Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:00.9295291Z 2022-03-17T19:41:00.9295587Z Running tests... 2022-03-17T19:41:00.9296245Z ---------------------------------------------------------------------- 2022-03-17T19:41:00.9317205Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-03-17T19:41:01.5203433Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27240 2022-03-17T19:41:01.5291319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27241 2022-03-17T19:41:02.4144033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:02.4351629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:04.1373511Z ok (3.207s) 2022-03-17T19:41:04.1373995Z 2022-03-17T19:41:04.1375932Z ---------------------------------------------------------------------- 2022-03-17T19:41:04.1376888Z Ran 1 test in 3.208s 2022-03-17T19:41:04.1377346Z 2022-03-17T19:41:04.1377614Z OK 2022-03-17T19:41:04.1377928Z 2022-03-17T19:41:04.1378220Z Generating XML reports... 2022-03-17T19:41:04.1459888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194100.xml 2022-03-17T19:41:06.6072621Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:06.6084233Z 2022-03-17T19:41:06.6084483Z Running tests... 2022-03-17T19:41:06.6084969Z ---------------------------------------------------------------------- 2022-03-17T19:41:07.1970368Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27516 2022-03-17T19:41:07.2006429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27517 2022-03-17T19:41:08.1075124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:08.1076086Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1079012Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1081641Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1084305Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1086787Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1089234Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1130825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:08.1134719Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1137413Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1140005Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1142454Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1144924Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:08.1147366Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:41:09.8081657Z ok (3.199s) 2022-03-17T19:41:09.8082327Z 2022-03-17T19:41:09.8083470Z ---------------------------------------------------------------------- 2022-03-17T19:41:09.8084415Z Ran 1 test in 3.200s 2022-03-17T19:41:09.8084851Z 2022-03-17T19:41:09.8085062Z OK 2022-03-17T19:41:09.8085354Z 2022-03-17T19:41:09.8085613Z Generating XML reports... 2022-03-17T19:41:09.8165042Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194106.xml 2022-03-17T19:41:12.1686445Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:12.1700714Z 2022-03-17T19:41:12.1701013Z Running tests... 2022-03-17T19:41:12.1701543Z ---------------------------------------------------------------------- 2022-03-17T19:41:12.7609968Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27720 2022-03-17T19:41:12.7643115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27721 2022-03-17T19:41:13.6560881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:13.6767228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:13.6978616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:41:13.6979745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:41:13.6980493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:41:13.6981270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:41:23.3792179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:41:23.3794531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:41:23.3853871Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:41:23.3856660Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:41:23.4056845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:41:23.4058152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:41:26.0946350Z ok (13.924s) 2022-03-17T19:41:26.0946894Z 2022-03-17T19:41:26.0948104Z ---------------------------------------------------------------------- 2022-03-17T19:41:26.0949167Z Ran 1 test in 13.924s 2022-03-17T19:41:26.0949638Z 2022-03-17T19:41:26.0949859Z OK 2022-03-17T19:41:26.0950226Z 2022-03-17T19:41:26.0950589Z Generating XML reports... 2022-03-17T19:41:26.1026289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194112.xml 2022-03-17T19:41:28.5518038Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:28.5533323Z 2022-03-17T19:41:28.5534136Z Running tests... 2022-03-17T19:41:28.5535524Z ---------------------------------------------------------------------- 2022-03-17T19:41:29.1548137Z test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27934 2022-03-17T19:41:29.1582698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27935 2022-03-17T19:41:30.0583961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:30.0646381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:31.9661874Z ok (3.413s) 2022-03-17T19:41:31.9662435Z 2022-03-17T19:41:31.9663548Z ---------------------------------------------------------------------- 2022-03-17T19:41:31.9664516Z Ran 1 test in 3.413s 2022-03-17T19:41:31.9664969Z 2022-03-17T19:41:31.9665169Z OK 2022-03-17T19:41:31.9665479Z 2022-03-17T19:41:31.9665764Z Generating XML reports... 2022-03-17T19:41:31.9739356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194128.xml 2022-03-17T19:41:34.3741990Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:34.3749471Z 2022-03-17T19:41:34.3750379Z Running tests... 2022-03-17T19:41:34.3751669Z ---------------------------------------------------------------------- 2022-03-17T19:41:34.9819101Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28210 2022-03-17T19:41:34.9879533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28211 2022-03-17T19:41:35.8505765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:35.8922310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:37.7964755Z ok (3.421s) 2022-03-17T19:41:37.7965254Z 2022-03-17T19:41:37.7966223Z ---------------------------------------------------------------------- 2022-03-17T19:41:37.7966996Z Ran 1 test in 3.421s 2022-03-17T19:41:37.7967383Z 2022-03-17T19:41:37.7967570Z OK 2022-03-17T19:41:37.7967884Z 2022-03-17T19:41:37.7968157Z Generating XML reports... 2022-03-17T19:41:37.8049279Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194134.xml 2022-03-17T19:41:40.2423680Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:40.2435542Z 2022-03-17T19:41:40.2435804Z Running tests... 2022-03-17T19:41:40.2436454Z ---------------------------------------------------------------------- 2022-03-17T19:41:40.8316419Z test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28486 2022-03-17T19:41:40.8350771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28487 2022-03-17T19:41:40.8398543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28488 2022-03-17T19:41:40.8444400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28489 2022-03-17T19:41:41.7290923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:41.7813089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:41.8360956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:41.8424692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:44.4547205Z ok (4.210s) 2022-03-17T19:41:44.4547669Z 2022-03-17T19:41:44.4549059Z ---------------------------------------------------------------------- 2022-03-17T19:41:44.4550071Z Ran 1 test in 4.211s 2022-03-17T19:41:44.4550556Z 2022-03-17T19:41:44.4550780Z OK 2022-03-17T19:41:44.4551152Z 2022-03-17T19:41:44.4551542Z Generating XML reports... 2022-03-17T19:41:44.4627356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194140.xml 2022-03-17T19:41:46.8831626Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:46.8844178Z 2022-03-17T19:41:46.8844485Z Running tests... 2022-03-17T19:41:46.8845419Z ---------------------------------------------------------------------- 2022-03-17T19:41:47.4738652Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28838 2022-03-17T19:41:47.4779012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28839 2022-03-17T19:41:47.4824377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28840 2022-03-17T19:41:47.4864853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28841 2022-03-17T19:41:48.4683212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:48.4770201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:48.5153316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:48.5258921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:50.7960436Z ok (3.911s) 2022-03-17T19:41:50.7960993Z 2022-03-17T19:41:50.7962069Z ---------------------------------------------------------------------- 2022-03-17T19:41:50.7962826Z Ran 1 test in 3.911s 2022-03-17T19:41:50.7963216Z 2022-03-17T19:41:50.7963401Z OK 2022-03-17T19:41:50.7965501Z 2022-03-17T19:41:50.7965960Z Generating XML reports... 2022-03-17T19:41:50.8041420Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194146.xml 2022-03-17T19:41:53.1628911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:53.1641033Z 2022-03-17T19:41:53.1641388Z Running tests... 2022-03-17T19:41:53.1642226Z ---------------------------------------------------------------------- 2022-03-17T19:41:53.7649007Z test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29190 2022-03-17T19:41:53.7688712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29191 2022-03-17T19:41:53.7740638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29192 2022-03-17T19:41:53.7786735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29193 2022-03-17T19:41:54.7313338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:54.7993171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:54.8040562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:54.8228494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:56.7886506Z ok (3.624s) 2022-03-17T19:41:56.7887083Z 2022-03-17T19:41:56.7888075Z ---------------------------------------------------------------------- 2022-03-17T19:41:56.7888856Z Ran 1 test in 3.624s 2022-03-17T19:41:56.7889255Z 2022-03-17T19:41:56.7889440Z OK 2022-03-17T19:41:56.7889742Z 2022-03-17T19:41:56.7890022Z Generating XML reports... 2022-03-17T19:41:56.7969656Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194153.xml 2022-03-17T19:41:59.3550065Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:41:59.3563175Z 2022-03-17T19:41:59.3563503Z Running tests... 2022-03-17T19:41:59.3564668Z ---------------------------------------------------------------------- 2022-03-17T19:41:59.9633604Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29542 2022-03-17T19:41:59.9675417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29543 2022-03-17T19:41:59.9759900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29544 2022-03-17T19:41:59.9824652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29545 2022-03-17T19:42:00.9337752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:00.9607404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:00.9778529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:00.9793565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:00.9951070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:42:01.0127975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:42:01.0129414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:42:01.0131609Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:01.0133021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:42:01.0134926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:01.0154756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:01.0230310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:03.6922802Z ok (4.336s) 2022-03-17T19:42:03.6923320Z 2022-03-17T19:42:03.6924794Z ---------------------------------------------------------------------- 2022-03-17T19:42:03.6925811Z Ran 1 test in 4.336s 2022-03-17T19:42:03.6926289Z 2022-03-17T19:42:03.6926534Z OK 2022-03-17T19:42:03.6926878Z 2022-03-17T19:42:03.6927230Z Generating XML reports... 2022-03-17T19:42:03.6995930Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194159.xml 2022-03-17T19:42:06.2680757Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:06.2689638Z 2022-03-17T19:42:06.2690256Z Running tests... 2022-03-17T19:42:06.2691324Z ---------------------------------------------------------------------- 2022-03-17T19:42:06.8760601Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29894 2022-03-17T19:42:06.8817717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29895 2022-03-17T19:42:06.8875974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29896 2022-03-17T19:42:06.8926112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29897 2022-03-17T19:42:07.8253662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:07.8446631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:07.8581436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:07.8650737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:09.8003561Z ok (3.531s) 2022-03-17T19:42:09.8004101Z 2022-03-17T19:42:09.8005219Z ---------------------------------------------------------------------- 2022-03-17T19:42:09.8006156Z Ran 1 test in 3.531s 2022-03-17T19:42:09.8006571Z 2022-03-17T19:42:09.8006791Z OK 2022-03-17T19:42:09.8007118Z 2022-03-17T19:42:09.8007502Z Generating XML reports... 2022-03-17T19:42:09.8085583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194206.xml 2022-03-17T19:42:12.3252023Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:12.3262567Z 2022-03-17T19:42:12.3263011Z Running tests... 2022-03-17T19:42:12.3264283Z ---------------------------------------------------------------------- 2022-03-17T19:42:12.9187998Z test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30246 2022-03-17T19:42:12.9226382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30247 2022-03-17T19:42:12.9275311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30248 2022-03-17T19:42:12.9314676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30249 2022-03-17T19:42:13.8569482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:13.9169158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:13.9458957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:13.9922894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:16.1401527Z ok (3.814s) 2022-03-17T19:42:16.1402205Z 2022-03-17T19:42:16.1403442Z ---------------------------------------------------------------------- 2022-03-17T19:42:16.1404357Z Ran 1 test in 3.814s 2022-03-17T19:42:16.1404728Z 2022-03-17T19:42:16.1404916Z OK 2022-03-17T19:42:16.1405211Z 2022-03-17T19:42:16.1405479Z Generating XML reports... 2022-03-17T19:42:16.1485719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194212.xml 2022-03-17T19:42:18.6423611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:18.6436657Z 2022-03-17T19:42:18.6436782Z Running tests... 2022-03-17T19:42:18.6437224Z ---------------------------------------------------------------------- 2022-03-17T19:42:19.2352610Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30622 2022-03-17T19:42:19.2385467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30623 2022-03-17T19:42:19.2425771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30624 2022-03-17T19:42:19.2465430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30625 2022-03-17T19:42:20.2334496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:20.2631405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:20.3326390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:20.3444201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:22.1553003Z skip (3.511s) 2022-03-17T19:42:22.1553533Z 2022-03-17T19:42:22.1554708Z ---------------------------------------------------------------------- 2022-03-17T19:42:22.1555639Z Ran 1 test in 3.511s 2022-03-17T19:42:22.1556046Z 2022-03-17T19:42:22.1556308Z OK (skipped=1) 2022-03-17T19:42:22.1556688Z 2022-03-17T19:42:22.1557016Z Generating XML reports... 2022-03-17T19:42:22.1621128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194218.xml 2022-03-17T19:42:24.5351588Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:24.5364229Z 2022-03-17T19:42:24.5364377Z Running tests... 2022-03-17T19:42:24.5364946Z ---------------------------------------------------------------------- 2022-03-17T19:42:25.1280951Z test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30962 2022-03-17T19:42:25.1325394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30963 2022-03-17T19:42:25.1385522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30964 2022-03-17T19:42:25.1440159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30965 2022-03-17T19:42:26.1385173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:26.1508070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:26.2424953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:26.2855512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:29.1549310Z ok (4.618s) 2022-03-17T19:42:29.1549807Z 2022-03-17T19:42:29.1551684Z ---------------------------------------------------------------------- 2022-03-17T19:42:29.1552520Z Ran 1 test in 4.618s 2022-03-17T19:42:29.1552887Z 2022-03-17T19:42:29.1553071Z OK 2022-03-17T19:42:29.1553346Z 2022-03-17T19:42:29.1553626Z Generating XML reports... 2022-03-17T19:42:29.1626452Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194224.xml 2022-03-17T19:42:31.6102364Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:31.6113382Z 2022-03-17T19:42:31.6113746Z Running tests... 2022-03-17T19:42:31.6114921Z ---------------------------------------------------------------------- 2022-03-17T19:42:32.2040573Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31314 2022-03-17T19:42:32.2080403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31315 2022-03-17T19:42:32.2131830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31316 2022-03-17T19:42:32.2174032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31317 2022-03-17T19:42:33.2061335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:33.2137817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:33.2614932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:33.2663612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:36.1275822Z ok (4.516s) 2022-03-17T19:42:36.1276463Z 2022-03-17T19:42:36.1277399Z ---------------------------------------------------------------------- 2022-03-17T19:42:36.1278206Z Ran 1 test in 4.516s 2022-03-17T19:42:36.1278573Z 2022-03-17T19:42:36.1278760Z OK 2022-03-17T19:42:36.1279036Z 2022-03-17T19:42:36.1279352Z Generating XML reports... 2022-03-17T19:42:36.1359083Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194231.xml 2022-03-17T19:42:38.5035850Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:38.5049834Z 2022-03-17T19:42:38.5050067Z Running tests... 2022-03-17T19:42:38.5050790Z ---------------------------------------------------------------------- 2022-03-17T19:42:39.0965047Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31666 2022-03-17T19:42:39.1005261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31667 2022-03-17T19:42:39.1054887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31668 2022-03-17T19:42:39.1101011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31669 2022-03-17T19:42:40.0689658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:40.0962727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:40.1322369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:40.2009113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:43.2220810Z ok (4.717s) 2022-03-17T19:42:43.2221415Z 2022-03-17T19:42:43.2222653Z ---------------------------------------------------------------------- 2022-03-17T19:42:43.2223417Z Ran 1 test in 4.717s 2022-03-17T19:42:43.2223793Z 2022-03-17T19:42:43.2223977Z OK 2022-03-17T19:42:43.2224264Z 2022-03-17T19:42:43.2224549Z Generating XML reports... 2022-03-17T19:42:43.2300754Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194238.xml 2022-03-17T19:42:45.6883282Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:45.6895296Z 2022-03-17T19:42:45.6895503Z Running tests... 2022-03-17T19:42:45.6896196Z ---------------------------------------------------------------------- 2022-03-17T19:42:46.2827927Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32018 2022-03-17T19:42:46.2863812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32019 2022-03-17T19:42:46.2913648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32020 2022-03-17T19:42:46.2954245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32021 2022-03-17T19:42:47.2090286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:47.2270624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:47.2694615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:47.3167355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:49.3053770Z ok (3.615s) 2022-03-17T19:42:49.3054309Z 2022-03-17T19:42:49.3055810Z ---------------------------------------------------------------------- 2022-03-17T19:42:49.3056841Z Ran 1 test in 3.616s 2022-03-17T19:42:49.3057389Z 2022-03-17T19:42:49.3057609Z OK 2022-03-17T19:42:49.3058008Z 2022-03-17T19:42:49.3058987Z Generating XML reports... 2022-03-17T19:42:49.3137036Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194245.xml 2022-03-17T19:42:51.7047384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:51.7060449Z 2022-03-17T19:42:51.7060845Z Running tests... 2022-03-17T19:42:51.7062413Z ---------------------------------------------------------------------- 2022-03-17T19:42:52.3002309Z test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32370 2022-03-17T19:42:52.3044792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32371 2022-03-17T19:42:52.3102661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32372 2022-03-17T19:42:52.3144143Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32373 2022-03-17T19:42:53.2897060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:53.2946738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:53.3127883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:53.3496857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:55.3290660Z ok (3.623s) 2022-03-17T19:42:55.3291150Z 2022-03-17T19:42:55.3292313Z ---------------------------------------------------------------------- 2022-03-17T19:42:55.3293113Z Ran 1 test in 3.623s 2022-03-17T19:42:55.3293482Z 2022-03-17T19:42:55.3293664Z OK 2022-03-17T19:42:55.3293950Z 2022-03-17T19:42:55.3294214Z Generating XML reports... 2022-03-17T19:42:55.3376300Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194251.xml 2022-03-17T19:42:57.8468214Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:42:57.8479687Z 2022-03-17T19:42:57.8479839Z Running tests... 2022-03-17T19:42:57.8480367Z ---------------------------------------------------------------------- 2022-03-17T19:42:58.4414833Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32722 2022-03-17T19:42:58.4459076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32723 2022-03-17T19:42:58.4521686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32724 2022-03-17T19:42:58.4565015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32725 2022-03-17T19:42:59.3598162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:59.4324485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:59.4393867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:59.4549753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:59.4660729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:42:59.4766288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:42:59.4863892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:42:59.4865956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:59.4867301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:42:59.4868978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:59.4871280Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:42:59.4873064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:43:01.4709551Z ok (3.623s) 2022-03-17T19:43:01.4709990Z 2022-03-17T19:43:01.4710890Z ---------------------------------------------------------------------- 2022-03-17T19:43:01.4711672Z Ran 1 test in 3.623s 2022-03-17T19:43:01.4712082Z 2022-03-17T19:43:01.4712263Z OK 2022-03-17T19:43:01.4712559Z 2022-03-17T19:43:01.4712840Z Generating XML reports... 2022-03-17T19:43:01.4796489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194257.xml 2022-03-17T19:43:03.9301388Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:03.9314491Z 2022-03-17T19:43:03.9314946Z Running tests... 2022-03-17T19:43:03.9315808Z ---------------------------------------------------------------------- 2022-03-17T19:43:04.5234851Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33074 2022-03-17T19:43:04.5273728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33075 2022-03-17T19:43:04.5324590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33076 2022-03-17T19:43:04.5365012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33077 2022-03-17T19:43:05.5485124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:05.5917210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:05.6064899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:05.6166786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:08.0479007Z ok (4.116s) 2022-03-17T19:43:08.0479414Z 2022-03-17T19:43:08.0480478Z ---------------------------------------------------------------------- 2022-03-17T19:43:08.0481730Z Ran 1 test in 4.116s 2022-03-17T19:43:08.0482391Z 2022-03-17T19:43:08.0482589Z OK 2022-03-17T19:43:08.0482985Z 2022-03-17T19:43:08.0483478Z Generating XML reports... 2022-03-17T19:43:08.0564463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194303.xml 2022-03-17T19:43:10.5522126Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:10.5532013Z 2022-03-17T19:43:10.5532648Z Running tests... 2022-03-17T19:43:10.5533807Z ---------------------------------------------------------------------- 2022-03-17T19:43:11.1468672Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33426 2022-03-17T19:43:11.1507354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33427 2022-03-17T19:43:11.1559078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33428 2022-03-17T19:43:11.1601322Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33429 2022-03-17T19:43:12.0922375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:12.1265208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:12.1407014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:12.2424465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:14.1708883Z ok (3.617s) 2022-03-17T19:43:14.1709268Z 2022-03-17T19:43:14.1710818Z ---------------------------------------------------------------------- 2022-03-17T19:43:14.1711586Z Ran 1 test in 3.617s 2022-03-17T19:43:14.1712463Z 2022-03-17T19:43:14.1712667Z OK 2022-03-17T19:43:14.1712959Z 2022-03-17T19:43:14.1713240Z Generating XML reports... 2022-03-17T19:43:14.1784166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194310.xml 2022-03-17T19:43:16.7239770Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:16.7251238Z 2022-03-17T19:43:16.7251493Z Running tests... 2022-03-17T19:43:16.7252019Z ---------------------------------------------------------------------- 2022-03-17T19:43:17.3174234Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33778 2022-03-17T19:43:17.3213432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33779 2022-03-17T19:43:17.3273239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33780 2022-03-17T19:43:17.3320989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33781 2022-03-17T19:43:18.3005975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:18.3317611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:18.3321013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:18.3433417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:20.4405399Z ok (3.715s) 2022-03-17T19:43:20.4405892Z 2022-03-17T19:43:20.4408849Z ---------------------------------------------------------------------- 2022-03-17T19:43:20.4409654Z Ran 1 test in 3.715s 2022-03-17T19:43:20.4410034Z 2022-03-17T19:43:20.4410224Z OK 2022-03-17T19:43:20.4410537Z 2022-03-17T19:43:20.4410857Z Generating XML reports... 2022-03-17T19:43:20.4487622Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194316.xml 2022-03-17T19:43:23.0847685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:23.0863187Z 2022-03-17T19:43:23.0863703Z Running tests... 2022-03-17T19:43:23.0864854Z ---------------------------------------------------------------------- 2022-03-17T19:43:23.6799595Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34130 2022-03-17T19:43:23.6838047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34131 2022-03-17T19:43:23.6887543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34132 2022-03-17T19:43:23.6932117Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34133 2022-03-17T19:43:24.6062028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:24.6739264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:24.7297870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:24.7485104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:27.5036800Z ok (4.417s) 2022-03-17T19:43:27.5037389Z 2022-03-17T19:43:27.5038544Z ---------------------------------------------------------------------- 2022-03-17T19:43:27.5039542Z Ran 1 test in 4.417s 2022-03-17T19:43:27.5040010Z 2022-03-17T19:43:27.5040249Z OK 2022-03-17T19:43:27.5040554Z 2022-03-17T19:43:27.5040865Z Generating XML reports... 2022-03-17T19:43:27.5119673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194323.xml 2022-03-17T19:43:30.0447146Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:30.0455084Z 2022-03-17T19:43:30.0455708Z Running tests... 2022-03-17T19:43:30.0457587Z ---------------------------------------------------------------------- 2022-03-17T19:43:30.6390794Z test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34506 2022-03-17T19:43:30.6427068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34507 2022-03-17T19:43:30.6481688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34508 2022-03-17T19:43:30.6529502Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34509 2022-03-17T19:43:31.6255180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:31.6617099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:31.6697975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:31.6942526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:33.7614300Z ok (3.715s) 2022-03-17T19:43:33.7614872Z 2022-03-17T19:43:33.7616455Z ---------------------------------------------------------------------- 2022-03-17T19:43:33.7617501Z Ran 1 test in 3.716s 2022-03-17T19:43:33.7617937Z 2022-03-17T19:43:33.7618159Z OK 2022-03-17T19:43:33.7618484Z 2022-03-17T19:43:33.7618801Z Generating XML reports... 2022-03-17T19:43:33.7697763Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194330.xml 2022-03-17T19:43:36.2570021Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:36.2581990Z 2022-03-17T19:43:36.2582383Z Running tests... 2022-03-17T19:43:36.2582842Z ---------------------------------------------------------------------- 2022-03-17T19:43:36.8514062Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34882 2022-03-17T19:43:36.8554385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34883 2022-03-17T19:43:36.8601123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34884 2022-03-17T19:43:36.8645935Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34885 2022-03-17T19:43:37.7975623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:37.8421362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:37.8422632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:37.8598174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:40.2733249Z skip (4.015s) 2022-03-17T19:43:40.2733859Z 2022-03-17T19:43:40.2735037Z ---------------------------------------------------------------------- 2022-03-17T19:43:40.2736036Z Ran 1 test in 4.015s 2022-03-17T19:43:40.2736384Z 2022-03-17T19:43:40.2736624Z OK (skipped=1) 2022-03-17T19:43:40.2736978Z 2022-03-17T19:43:40.2737247Z Generating XML reports... 2022-03-17T19:43:40.2817536Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194336.xml 2022-03-17T19:43:42.7169741Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:42.7182957Z 2022-03-17T19:43:42.7183156Z Running tests... 2022-03-17T19:43:42.7183586Z ---------------------------------------------------------------------- 2022-03-17T19:43:43.3130708Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35222 2022-03-17T19:43:43.3179617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35223 2022-03-17T19:43:43.3235469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 35224 2022-03-17T19:43:43.3278112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 35225 2022-03-17T19:43:44.2870351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:44.2917328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:44.2950847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:44.2992904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:46.4377635Z ok (3.719s) 2022-03-17T19:43:46.4378002Z 2022-03-17T19:43:46.4379839Z ---------------------------------------------------------------------- 2022-03-17T19:43:46.4380849Z Ran 1 test in 3.719s 2022-03-17T19:43:46.4381306Z 2022-03-17T19:43:46.4381544Z OK 2022-03-17T19:43:46.4381911Z 2022-03-17T19:43:46.4382281Z Generating XML reports... 2022-03-17T19:43:46.4460106Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194342.xml 2022-03-17T19:43:48.8185953Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:48.8197019Z 2022-03-17T19:43:48.8197608Z Running tests... 2022-03-17T19:43:48.8198807Z ---------------------------------------------------------------------- 2022-03-17T19:43:49.4085706Z test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35574 2022-03-17T19:43:49.4131235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35575 2022-03-17T19:43:49.4162225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 35576 2022-03-17T19:43:49.4191633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 35577 2022-03-17T19:43:50.3972880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:50.4024924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:50.4069661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:50.4106791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:52.6291290Z ok (3.809s) 2022-03-17T19:43:52.6291724Z 2022-03-17T19:43:52.6292989Z ---------------------------------------------------------------------- 2022-03-17T19:43:52.6294676Z Ran 1 test in 3.809s 2022-03-17T19:43:52.6295091Z 2022-03-17T19:43:52.6295277Z OK 2022-03-17T19:43:52.6299097Z 2022-03-17T19:43:52.6299884Z Generating XML reports... 2022-03-17T19:43:52.6372420Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194348.xml 2022-03-17T19:43:55.1169684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:43:55.1180798Z 2022-03-17T19:43:55.1181296Z Running tests... 2022-03-17T19:43:55.7085999Z ---------------------------------------------------------------------- 2022-03-17T19:43:55.7087392Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35926 2022-03-17T19:43:55.7139475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35927 2022-03-17T19:43:55.7197884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 35928 2022-03-17T19:43:55.7245737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 35929 2022-03-17T19:43:56.6723761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:56.6912837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:56.6971438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:56.7106183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:59.1352978Z ok (4.017s) 2022-03-17T19:43:59.1353439Z 2022-03-17T19:43:59.1354590Z ---------------------------------------------------------------------- 2022-03-17T19:43:59.1355564Z Ran 1 test in 4.017s 2022-03-17T19:43:59.1355993Z 2022-03-17T19:43:59.1356251Z OK 2022-03-17T19:43:59.1356616Z 2022-03-17T19:43:59.1356976Z Generating XML reports... 2022-03-17T19:43:59.1435701Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194355.xml 2022-03-17T19:44:01.6048153Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:01.6058538Z 2022-03-17T19:44:01.6059011Z Running tests... 2022-03-17T19:44:01.6060461Z ---------------------------------------------------------------------- 2022-03-17T19:44:02.2005232Z test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36278 2022-03-17T19:44:02.2043950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36279 2022-03-17T19:44:02.2095247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36280 2022-03-17T19:44:02.2139154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36281 2022-03-17T19:44:03.1917975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:03.2234897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:03.2577345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:03.2720394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:05.1219293Z ok (3.516s) 2022-03-17T19:44:05.1219696Z 2022-03-17T19:44:05.1220806Z ---------------------------------------------------------------------- 2022-03-17T19:44:05.1221661Z Ran 1 test in 3.516s 2022-03-17T19:44:05.1222045Z 2022-03-17T19:44:05.1222232Z OK 2022-03-17T19:44:05.1222544Z 2022-03-17T19:44:05.1222828Z Generating XML reports... 2022-03-17T19:44:05.1298265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194401.xml 2022-03-17T19:44:07.5681439Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:07.5692862Z 2022-03-17T19:44:07.5693419Z Running tests... 2022-03-17T19:44:07.5693978Z ---------------------------------------------------------------------- 2022-03-17T19:44:08.1615195Z test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36630 2022-03-17T19:44:08.1655424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36631 2022-03-17T19:44:08.1705149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36632 2022-03-17T19:44:08.1749782Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36633 2022-03-17T19:44:09.1117019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:09.1333133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:09.1882184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:09.1949980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:11.5852537Z ok (4.015s) 2022-03-17T19:44:11.5852746Z 2022-03-17T19:44:11.5853291Z ---------------------------------------------------------------------- 2022-03-17T19:44:11.5853678Z Ran 1 test in 4.016s 2022-03-17T19:44:11.5853859Z 2022-03-17T19:44:11.5853990Z OK 2022-03-17T19:44:11.5854139Z 2022-03-17T19:44:11.5854284Z Generating XML reports... 2022-03-17T19:44:11.5938255Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194407.xml 2022-03-17T19:44:14.0448801Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:14.0461045Z 2022-03-17T19:44:14.0461323Z Running tests... 2022-03-17T19:44:14.0461857Z ---------------------------------------------------------------------- 2022-03-17T19:44:14.6295636Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... skip (0.583s) 2022-03-17T19:44:14.6296453Z 2022-03-17T19:44:14.6297492Z ---------------------------------------------------------------------- 2022-03-17T19:44:14.6298614Z Ran 1 test in 0.583s 2022-03-17T19:44:14.6298987Z 2022-03-17T19:44:14.6299224Z OK (skipped=1) 2022-03-17T19:44:14.6299572Z 2022-03-17T19:44:14.6299860Z Generating XML reports... 2022-03-17T19:44:14.6326442Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194414.xml 2022-03-17T19:44:17.0418428Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:17.0431379Z 2022-03-17T19:44:17.0431696Z Running tests... 2022-03-17T19:44:17.0432221Z ---------------------------------------------------------------------- 2022-03-17T19:44:17.6386354Z test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37073 2022-03-17T19:44:17.6423850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37074 2022-03-17T19:44:17.6479785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37075 2022-03-17T19:44:17.6530045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37076 2022-03-17T19:44:18.6144478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:18.6608783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:18.6806636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:18.6992696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:21.1690610Z ok (4.125s) 2022-03-17T19:44:21.1691137Z 2022-03-17T19:44:21.1692362Z ---------------------------------------------------------------------- 2022-03-17T19:44:21.1693148Z Ran 1 test in 4.126s 2022-03-17T19:44:21.1693520Z 2022-03-17T19:44:21.1693688Z OK 2022-03-17T19:44:21.1693976Z 2022-03-17T19:44:21.1694927Z Generating XML reports... 2022-03-17T19:44:21.1777443Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194417.xml 2022-03-17T19:44:23.9382842Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:23.9390790Z 2022-03-17T19:44:23.9391046Z Running tests... 2022-03-17T19:44:23.9392143Z ---------------------------------------------------------------------- 2022-03-17T19:44:24.5330942Z test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37425 2022-03-17T19:44:24.5369414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37426 2022-03-17T19:44:24.5425645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37427 2022-03-17T19:44:24.5471826Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37428 2022-03-17T19:44:25.5184053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:25.5197434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:25.5995615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:25.6127381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:27.6558644Z ok (3.716s) 2022-03-17T19:44:27.6559229Z 2022-03-17T19:44:27.6561157Z ---------------------------------------------------------------------- 2022-03-17T19:44:27.6562328Z Ran 1 test in 3.716s 2022-03-17T19:44:27.6562704Z 2022-03-17T19:44:27.6562892Z OK 2022-03-17T19:44:27.6563185Z 2022-03-17T19:44:27.6563451Z Generating XML reports... 2022-03-17T19:44:27.6639407Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194423.xml 2022-03-17T19:44:30.1289899Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:30.1301363Z 2022-03-17T19:44:30.1301614Z Running tests... 2022-03-17T19:44:30.7249640Z ---------------------------------------------------------------------- 2022-03-17T19:44:30.7250241Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37777 2022-03-17T19:44:30.7299114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37778 2022-03-17T19:44:30.7358257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37779 2022-03-17T19:44:30.7413430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37780 2022-03-17T19:44:31.7009451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:31.7896910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:31.8260867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:31.8436845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:34.9539106Z ok (4.823s) 2022-03-17T19:44:34.9539501Z 2022-03-17T19:44:34.9540788Z ---------------------------------------------------------------------- 2022-03-17T19:44:34.9541546Z Ran 1 test in 4.824s 2022-03-17T19:44:34.9541926Z 2022-03-17T19:44:34.9542111Z OK 2022-03-17T19:44:34.9542400Z 2022-03-17T19:44:34.9542703Z Generating XML reports... 2022-03-17T19:44:34.9622277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194430.xml 2022-03-17T19:44:37.4954430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:37.4970008Z 2022-03-17T19:44:37.4970447Z Running tests... 2022-03-17T19:44:37.4970981Z ---------------------------------------------------------------------- 2022-03-17T19:44:38.0925320Z test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38129 2022-03-17T19:44:38.0965731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38130 2022-03-17T19:44:38.1062800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38131 2022-03-17T19:44:38.1131965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38132 2022-03-17T19:44:39.0472740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:39.0521264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:39.0624421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:39.0717874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:41.5240325Z ok (4.027s) 2022-03-17T19:44:41.5240743Z 2022-03-17T19:44:41.5242200Z ---------------------------------------------------------------------- 2022-03-17T19:44:41.5243196Z Ran 1 test in 4.027s 2022-03-17T19:44:41.5243657Z 2022-03-17T19:44:41.5243921Z OK 2022-03-17T19:44:41.5244250Z 2022-03-17T19:44:41.5244537Z Generating XML reports... 2022-03-17T19:44:41.5327922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194437.xml 2022-03-17T19:44:44.0424061Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:44.0434985Z 2022-03-17T19:44:44.0435340Z Running tests... 2022-03-17T19:44:44.0435859Z ---------------------------------------------------------------------- 2022-03-17T19:44:44.6366576Z test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38481 2022-03-17T19:44:44.6407110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38482 2022-03-17T19:44:44.6460516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38483 2022-03-17T19:44:44.6506344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38484 2022-03-17T19:44:45.6178899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:45.6635570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:45.6872871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:45.6933494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:47.9584092Z ok (3.914s) 2022-03-17T19:44:47.9584559Z 2022-03-17T19:44:47.9585635Z ---------------------------------------------------------------------- 2022-03-17T19:44:47.9586398Z Ran 1 test in 3.915s 2022-03-17T19:44:47.9586764Z 2022-03-17T19:44:47.9586957Z OK 2022-03-17T19:44:47.9587255Z 2022-03-17T19:44:47.9590585Z Generating XML reports... 2022-03-17T19:44:47.9666867Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194444.xml 2022-03-17T19:44:50.4718231Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:50.4728423Z 2022-03-17T19:44:50.4729013Z Running tests... 2022-03-17T19:44:50.4730416Z ---------------------------------------------------------------------- 2022-03-17T19:44:51.0660657Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38857 2022-03-17T19:44:51.0699444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38858 2022-03-17T19:44:51.0746858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38859 2022-03-17T19:44:51.0786876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38860 2022-03-17T19:44:52.0317428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:52.0458962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:52.0501270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:52.0775795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:54.0953851Z skip (3.622s) 2022-03-17T19:44:54.0955269Z 2022-03-17T19:44:54.0956715Z ---------------------------------------------------------------------- 2022-03-17T19:44:54.0957753Z Ran 1 test in 3.622s 2022-03-17T19:44:54.0958131Z 2022-03-17T19:44:54.0958346Z OK (skipped=1) 2022-03-17T19:44:54.0958699Z 2022-03-17T19:44:54.0958971Z Generating XML reports... 2022-03-17T19:44:54.1038400Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194450.xml 2022-03-17T19:44:56.5540728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:44:56.5552065Z 2022-03-17T19:44:56.5552401Z Running tests... 2022-03-17T19:44:56.5553585Z ---------------------------------------------------------------------- 2022-03-17T19:44:57.1490582Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39197 2022-03-17T19:44:57.1534828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39198 2022-03-17T19:44:57.1605967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39199 2022-03-17T19:44:57.1649475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39200 2022-03-17T19:44:58.0994468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:58.1458699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:58.1499447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:58.1897393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:00.7744401Z ok (4.218s) 2022-03-17T19:45:00.7745416Z 2022-03-17T19:45:00.7747141Z ---------------------------------------------------------------------- 2022-03-17T19:45:00.7748383Z Ran 1 test in 4.219s 2022-03-17T19:45:00.7748918Z 2022-03-17T19:45:00.7749244Z OK 2022-03-17T19:45:00.7749553Z 2022-03-17T19:45:00.7750090Z Generating XML reports... 2022-03-17T19:45:00.7826522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194456.xml 2022-03-17T19:45:03.2981276Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:03.2991455Z 2022-03-17T19:45:03.2992743Z Running tests... 2022-03-17T19:45:03.2994331Z ---------------------------------------------------------------------- 2022-03-17T19:45:03.8919788Z test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39553 2022-03-17T19:45:03.8956618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39554 2022-03-17T19:45:03.9010502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39555 2022-03-17T19:45:03.9055660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39556 2022-03-17T19:45:04.8189688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:04.8190403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:04.8684769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:04.8837410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:07.6147414Z ok (4.315s) 2022-03-17T19:45:07.6148296Z 2022-03-17T19:45:07.6150354Z ---------------------------------------------------------------------- 2022-03-17T19:45:07.6151449Z Ran 1 test in 4.316s 2022-03-17T19:45:07.6152002Z 2022-03-17T19:45:07.6152346Z OK 2022-03-17T19:45:07.6152850Z 2022-03-17T19:45:07.6153303Z Generating XML reports... 2022-03-17T19:45:07.6226640Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194503.xml 2022-03-17T19:45:10.1361676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:10.1371043Z 2022-03-17T19:45:10.1371694Z Running tests... 2022-03-17T19:45:10.1374164Z ---------------------------------------------------------------------- 2022-03-17T19:45:10.7291817Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39905 2022-03-17T19:45:10.7329362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39906 2022-03-17T19:45:10.7392578Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 39907 2022-03-17T19:45:10.7432456Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 39908 2022-03-17T19:45:11.7240393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:11.7546015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:11.7717973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:11.7780921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:15.7560578Z ok (5.618s) 2022-03-17T19:45:15.7561189Z 2022-03-17T19:45:15.7565080Z ---------------------------------------------------------------------- 2022-03-17T19:45:15.7566147Z Ran 1 test in 5.619s 2022-03-17T19:45:15.7566635Z 2022-03-17T19:45:15.7566807Z OK 2022-03-17T19:45:15.7567145Z 2022-03-17T19:45:15.7567474Z Generating XML reports... 2022-03-17T19:45:15.7642979Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194510.xml 2022-03-17T19:45:18.4456218Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:18.4468191Z 2022-03-17T19:45:18.4468391Z Running tests... 2022-03-17T19:45:18.4469064Z ---------------------------------------------------------------------- 2022-03-17T19:45:19.0430056Z test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40257 2022-03-17T19:45:19.0469238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40258 2022-03-17T19:45:19.0522984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40259 2022-03-17T19:45:19.0572890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40260 2022-03-17T19:45:19.9711161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:19.9783820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:20.0177982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:20.0767966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:22.0662138Z ok (3.619s) 2022-03-17T19:45:22.0662971Z 2022-03-17T19:45:22.0664576Z ---------------------------------------------------------------------- 2022-03-17T19:45:22.0666112Z Ran 1 test in 3.619s 2022-03-17T19:45:22.0666492Z 2022-03-17T19:45:22.0666844Z OK 2022-03-17T19:45:22.0667355Z 2022-03-17T19:45:22.0668972Z Generating XML reports... 2022-03-17T19:45:22.0742608Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194518.xml 2022-03-17T19:45:24.5831766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:24.5843185Z 2022-03-17T19:45:24.5843731Z Running tests... 2022-03-17T19:45:24.5880865Z ---------------------------------------------------------------------- 2022-03-17T19:45:25.1763752Z test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40609 2022-03-17T19:45:25.1799079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40610 2022-03-17T19:45:25.1857090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40611 2022-03-17T19:45:25.1905106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40612 2022-03-17T19:45:26.0916121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:26.1793966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:26.1867259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:26.1871591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:28.6007317Z ok (4.016s) 2022-03-17T19:45:28.6007904Z 2022-03-17T19:45:28.6009133Z ---------------------------------------------------------------------- 2022-03-17T19:45:28.6010358Z Ran 1 test in 4.016s 2022-03-17T19:45:28.6010937Z 2022-03-17T19:45:28.6011265Z OK 2022-03-17T19:45:28.6011661Z 2022-03-17T19:45:28.6012037Z Generating XML reports... 2022-03-17T19:45:28.6075447Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194524.xml 2022-03-17T19:45:31.0470268Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:31.0480394Z 2022-03-17T19:45:31.0480989Z Running tests... 2022-03-17T19:45:31.0482616Z ---------------------------------------------------------------------- 2022-03-17T19:45:31.6499125Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40985 2022-03-17T19:45:31.6532563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40986 2022-03-17T19:45:31.6576055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40987 2022-03-17T19:45:31.6610622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40988 2022-03-17T19:45:32.5793598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:32.6432637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:32.6516602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:32.6600715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:34.4685032Z skip (3.420s) 2022-03-17T19:45:34.4685461Z 2022-03-17T19:45:34.4686670Z ---------------------------------------------------------------------- 2022-03-17T19:45:34.4687666Z Ran 1 test in 3.420s 2022-03-17T19:45:34.4688162Z 2022-03-17T19:45:34.4688524Z OK (skipped=1) 2022-03-17T19:45:34.4688986Z 2022-03-17T19:45:34.4689462Z Generating XML reports... 2022-03-17T19:45:34.4769795Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194531.xml 2022-03-17T19:45:36.9244736Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:36.9257355Z 2022-03-17T19:45:36.9257657Z Running tests... 2022-03-17T19:45:36.9258110Z ---------------------------------------------------------------------- 2022-03-17T19:45:37.5181887Z test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41325 2022-03-17T19:45:37.5228929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41326 2022-03-17T19:45:37.5290333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41327 2022-03-17T19:45:37.5331232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41328 2022-03-17T19:45:38.4805470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:38.4974844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:38.5117049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:38.6059798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:40.6437868Z ok (3.718s) 2022-03-17T19:45:40.6438550Z 2022-03-17T19:45:40.6440081Z ---------------------------------------------------------------------- 2022-03-17T19:45:40.6441163Z Ran 1 test in 3.718s 2022-03-17T19:45:40.6441624Z 2022-03-17T19:45:40.6442155Z OK 2022-03-17T19:45:40.6442606Z 2022-03-17T19:45:40.6443040Z Generating XML reports... 2022-03-17T19:45:40.6520795Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194536.xml 2022-03-17T19:45:43.1351107Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:43.1361105Z 2022-03-17T19:45:43.1361811Z Running tests... 2022-03-17T19:45:43.1363731Z ---------------------------------------------------------------------- 2022-03-17T19:45:43.7307876Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41689 2022-03-17T19:45:43.7344753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41690 2022-03-17T19:45:43.7395587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41691 2022-03-17T19:45:43.7436817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41692 2022-03-17T19:45:44.7528536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:44.7684116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:44.7688141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:44.7713929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:47.6524985Z ok (4.516s) 2022-03-17T19:45:47.6525427Z 2022-03-17T19:45:47.6526421Z ---------------------------------------------------------------------- 2022-03-17T19:45:47.6527604Z Ran 1 test in 4.516s 2022-03-17T19:45:47.6579764Z 2022-03-17T19:45:47.6580010Z OK 2022-03-17T19:45:47.6580328Z 2022-03-17T19:45:47.6580646Z Generating XML reports... 2022-03-17T19:45:47.6610482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194543.xml 2022-03-17T19:45:50.1424676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:50.1435360Z 2022-03-17T19:45:50.1436059Z Running tests... 2022-03-17T19:45:50.1437248Z ---------------------------------------------------------------------- 2022-03-17T19:45:50.7345165Z test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42077 2022-03-17T19:45:50.7381999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42078 2022-03-17T19:45:50.7434818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42079 2022-03-17T19:45:50.7486178Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42080 2022-03-17T19:45:51.6635688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:51.7904538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:51.7905707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:51.8160116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:54.3590775Z ok (4.215s) 2022-03-17T19:45:54.3591253Z 2022-03-17T19:45:54.3592436Z ---------------------------------------------------------------------- 2022-03-17T19:45:54.3593541Z Ran 1 test in 4.215s 2022-03-17T19:45:54.3594058Z 2022-03-17T19:45:54.3594299Z OK 2022-03-17T19:45:54.3594574Z 2022-03-17T19:45:54.3594879Z Generating XML reports... 2022-03-17T19:45:54.3668917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194550.xml 2022-03-17T19:45:56.8708733Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:45:56.8720025Z 2022-03-17T19:45:56.8720345Z Running tests... 2022-03-17T19:45:56.8720801Z ---------------------------------------------------------------------- 2022-03-17T19:45:57.4638650Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42429 2022-03-17T19:45:57.4676775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42430 2022-03-17T19:45:57.4727371Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42431 2022-03-17T19:45:57.4768206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42432 2022-03-17T19:45:58.3898270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:58.4155724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:58.4609621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:58.4616300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:00.6862654Z ok (3.814s) 2022-03-17T19:46:00.6863447Z 2022-03-17T19:46:00.6864692Z ---------------------------------------------------------------------- 2022-03-17T19:46:00.6865555Z Ran 1 test in 3.814s 2022-03-17T19:46:00.6865941Z 2022-03-17T19:46:00.6866172Z OK 2022-03-17T19:46:00.6866501Z 2022-03-17T19:46:00.6866818Z Generating XML reports... 2022-03-17T19:46:00.6896083Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194556.xml 2022-03-17T19:46:03.1409475Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:03.1423608Z 2022-03-17T19:46:03.1423940Z Running tests... 2022-03-17T19:46:03.1424483Z ---------------------------------------------------------------------- 2022-03-17T19:46:03.7396084Z test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42781 2022-03-17T19:46:03.7440800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42782 2022-03-17T19:46:03.7494371Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42783 2022-03-17T19:46:03.7590553Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42784 2022-03-17T19:46:04.7572580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:04.7755976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:04.7795411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:04.7817968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:06.6630432Z ok (3.520s) 2022-03-17T19:46:06.6630867Z 2022-03-17T19:46:06.6631936Z ---------------------------------------------------------------------- 2022-03-17T19:46:06.6632803Z Ran 1 test in 3.521s 2022-03-17T19:46:06.6633214Z 2022-03-17T19:46:06.6633435Z OK 2022-03-17T19:46:06.6633720Z 2022-03-17T19:46:06.6634027Z Generating XML reports... 2022-03-17T19:46:06.6705697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194603.xml 2022-03-17T19:46:09.1721465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:09.1733367Z 2022-03-17T19:46:09.1734417Z Running tests... 2022-03-17T19:46:09.1736119Z ---------------------------------------------------------------------- 2022-03-17T19:46:09.7666253Z test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43133 2022-03-17T19:46:09.7701263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43134 2022-03-17T19:46:09.7754389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43135 2022-03-17T19:46:09.7802800Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43136 2022-03-17T19:46:10.7734373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:10.7839086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:10.8043542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:10.8421197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:13.0909799Z ok (3.917s) 2022-03-17T19:46:13.0910166Z 2022-03-17T19:46:13.0913331Z ---------------------------------------------------------------------- 2022-03-17T19:46:13.0914794Z Ran 1 test in 3.918s 2022-03-17T19:46:13.0915214Z 2022-03-17T19:46:13.0915424Z OK 2022-03-17T19:46:13.0915745Z 2022-03-17T19:46:13.0916006Z Generating XML reports... 2022-03-17T19:46:13.0995572Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194609.xml 2022-03-17T19:46:15.5940062Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:15.5951985Z 2022-03-17T19:46:15.5952258Z Running tests... 2022-03-17T19:46:15.5952721Z ---------------------------------------------------------------------- 2022-03-17T19:46:15.5959071Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip (0.001s) 2022-03-17T19:46:15.5959476Z 2022-03-17T19:46:15.5959848Z ---------------------------------------------------------------------- 2022-03-17T19:46:15.5960203Z Ran 1 test in 0.001s 2022-03-17T19:46:15.5960383Z 2022-03-17T19:46:15.5960507Z OK (skipped=1) 2022-03-17T19:46:15.5960681Z 2022-03-17T19:46:15.5960818Z Generating XML reports... 2022-03-17T19:46:15.5992540Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194615.xml 2022-03-17T19:46:18.2348775Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:18.2358868Z 2022-03-17T19:46:18.2359193Z Running tests... 2022-03-17T19:46:18.2360707Z ---------------------------------------------------------------------- 2022-03-17T19:46:18.8296178Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43576 2022-03-17T19:46:18.8345818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43577 2022-03-17T19:46:18.8418460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43578 2022-03-17T19:46:18.8459853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43579 2022-03-17T19:46:19.7733868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:19.7844704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:19.7846590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:19.8041428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:21.7561502Z ok (3.520s) 2022-03-17T19:46:21.7562550Z 2022-03-17T19:46:21.7563835Z ---------------------------------------------------------------------- 2022-03-17T19:46:21.7564757Z Ran 1 test in 3.520s 2022-03-17T19:46:21.7565147Z 2022-03-17T19:46:21.7565373Z OK 2022-03-17T19:46:21.7565713Z 2022-03-17T19:46:21.7566048Z Generating XML reports... 2022-03-17T19:46:21.7649649Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194618.xml 2022-03-17T19:46:24.1549083Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:24.1561229Z 2022-03-17T19:46:24.1561580Z Running tests... 2022-03-17T19:46:24.1562413Z ---------------------------------------------------------------------- 2022-03-17T19:46:24.1566078Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip (0.000s) 2022-03-17T19:46:24.1566447Z 2022-03-17T19:46:24.1566932Z ---------------------------------------------------------------------- 2022-03-17T19:46:24.1567435Z Ran 1 test in 0.001s 2022-03-17T19:46:24.1567710Z 2022-03-17T19:46:24.1567946Z OK (skipped=1) 2022-03-17T19:46:24.1568122Z 2022-03-17T19:46:24.1568262Z Generating XML reports... 2022-03-17T19:46:24.1600727Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194624.xml 2022-03-17T19:46:26.6626799Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:26.6639615Z 2022-03-17T19:46:26.6639961Z Running tests... 2022-03-17T19:46:26.6640600Z ---------------------------------------------------------------------- 2022-03-17T19:46:27.2578949Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43995 2022-03-17T19:46:27.2612996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43996 2022-03-17T19:46:27.2664776Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43997 2022-03-17T19:46:27.2708363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43998 2022-03-17T19:46:28.2167761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:28.2382453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:28.2454585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:28.2463943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:31.6831663Z ok (5.019s) 2022-03-17T19:46:31.6832123Z 2022-03-17T19:46:31.6833355Z ---------------------------------------------------------------------- 2022-03-17T19:46:31.6834426Z Ran 1 test in 5.019s 2022-03-17T19:46:31.6834921Z 2022-03-17T19:46:31.6835139Z OK 2022-03-17T19:46:31.6835544Z 2022-03-17T19:46:31.6835940Z Generating XML reports... 2022-03-17T19:46:31.6917853Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194626.xml 2022-03-17T19:46:34.0802118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:34.0815637Z 2022-03-17T19:46:34.0815821Z Running tests... 2022-03-17T19:46:34.0816377Z ---------------------------------------------------------------------- 2022-03-17T19:46:34.6767249Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44719 2022-03-17T19:46:34.6811256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44720 2022-03-17T19:46:34.6863977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44721 2022-03-17T19:46:34.6911630Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44722 2022-03-17T19:46:35.6079046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:35.6291261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:35.6875595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:35.7339370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:38.1020275Z ok (4.020s) 2022-03-17T19:46:38.1020963Z 2022-03-17T19:46:38.1022037Z ---------------------------------------------------------------------- 2022-03-17T19:46:38.1022909Z Ran 1 test in 4.020s 2022-03-17T19:46:38.1023334Z 2022-03-17T19:46:38.1023543Z OK 2022-03-17T19:46:38.1023858Z 2022-03-17T19:46:38.1024173Z Generating XML reports... 2022-03-17T19:46:38.1103631Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194634.xml 2022-03-17T19:46:40.5094699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:40.5107963Z 2022-03-17T19:46:40.5108177Z Running tests... 2022-03-17T19:46:40.5108856Z ---------------------------------------------------------------------- 2022-03-17T19:46:40.5232821Z test_forward_backward (__main__.ReducerTest) ... ok (0.012s) 2022-03-17T19:46:40.5335082Z 2022-03-17T19:46:40.5335810Z ---------------------------------------------------------------------- 2022-03-17T19:46:40.5336273Z Ran 1 test in 0.023s 2022-03-17T19:46:40.5336504Z 2022-03-17T19:46:40.5336962Z OK 2022-03-17T19:46:40.5337121Z 2022-03-17T19:46:40.5337260Z Generating XML reports... 2022-03-17T19:46:40.5369455Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194640.xml 2022-03-17T19:46:43.1339698Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:43.1348957Z 2022-03-17T19:46:43.1349305Z Running tests... 2022-03-17T19:46:43.1350831Z ---------------------------------------------------------------------- 2022-03-17T19:46:43.1498322Z test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:46:43.1540407Z ok (0.019s) 2022-03-17T19:46:43.1578311Z 2022-03-17T19:46:43.1579513Z ---------------------------------------------------------------------- 2022-03-17T19:46:43.1580566Z Ran 1 test in 0.023s 2022-03-17T19:46:43.1581079Z 2022-03-17T19:46:43.1581369Z OK 2022-03-17T19:46:43.1581709Z 2022-03-17T19:46:43.1581983Z Generating XML reports... 2022-03-17T19:46:43.1614241Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194643.xml 2022-03-17T19:46:45.6167264Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:45.6176909Z 2022-03-17T19:46:45.6177581Z Running tests... 2022-03-17T19:46:45.6178808Z ---------------------------------------------------------------------- 2022-03-17T19:46:45.6307100Z test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.013s) 2022-03-17T19:46:45.6405898Z 2022-03-17T19:46:45.6407109Z ---------------------------------------------------------------------- 2022-03-17T19:46:45.6408176Z Ran 1 test in 0.023s 2022-03-17T19:46:45.6408668Z 2022-03-17T19:46:45.6408944Z OK 2022-03-17T19:46:45.6409340Z 2022-03-17T19:46:45.6409732Z Generating XML reports... 2022-03-17T19:46:45.6439481Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194645.xml 2022-03-17T19:46:48.2766113Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:48.2778164Z 2022-03-17T19:46:48.2778412Z Running tests... 2022-03-17T19:46:48.2779101Z ---------------------------------------------------------------------- 2022-03-17T19:46:48.2858788Z test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.008s) 2022-03-17T19:46:48.2898639Z 2022-03-17T19:46:48.2899686Z ---------------------------------------------------------------------- 2022-03-17T19:46:48.2900778Z Ran 1 test in 0.012s 2022-03-17T19:46:48.2901144Z 2022-03-17T19:46:48.2901354Z OK 2022-03-17T19:46:48.2901695Z 2022-03-17T19:46:48.2902110Z Generating XML reports... 2022-03-17T19:46:48.2933652Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194648.xml 2022-03-17T19:46:51.5078407Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:51.5088692Z 2022-03-17T19:46:51.5089675Z Running tests... 2022-03-17T19:46:51.5090998Z ---------------------------------------------------------------------- 2022-03-17T19:46:51.5252995Z test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.016s) 2022-03-17T19:46:51.5308905Z 2022-03-17T19:46:51.5311897Z ---------------------------------------------------------------------- 2022-03-17T19:46:51.5312798Z Ran 1 test in 0.022s 2022-03-17T19:46:51.5313205Z 2022-03-17T19:46:51.5313428Z OK 2022-03-17T19:46:51.5313760Z 2022-03-17T19:46:51.5314067Z Generating XML reports... 2022-03-17T19:46:51.5352909Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194651.xml 2022-03-17T19:46:54.1197226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:54.1207897Z 2022-03-17T19:46:54.1208417Z Running tests... 2022-03-17T19:46:54.1209198Z ---------------------------------------------------------------------- 2022-03-17T19:46:54.1277770Z test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.007s) 2022-03-17T19:46:54.1327814Z 2022-03-17T19:46:54.1328354Z ---------------------------------------------------------------------- 2022-03-17T19:46:54.1328842Z Ran 1 test in 0.012s 2022-03-17T19:46:54.1329096Z 2022-03-17T19:46:54.1329230Z OK 2022-03-17T19:46:54.1329422Z 2022-03-17T19:46:54.1329603Z Generating XML reports... 2022-03-17T19:46:54.1362852Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194654.xml 2022-03-17T19:46:56.6974074Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:56.6984833Z 2022-03-17T19:46:56.6985383Z Running tests... 2022-03-17T19:46:56.6986672Z ---------------------------------------------------------------------- 2022-03-17T19:46:57.2848201Z test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:46:57.2850329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:46:57.2933921Z ok (0.595s) 2022-03-17T19:46:57.2934613Z 2022-03-17T19:46:57.2986976Z ---------------------------------------------------------------------- 2022-03-17T19:46:57.2987916Z Ran 1 test in 0.595s 2022-03-17T19:46:57.2988316Z 2022-03-17T19:46:57.2988519Z OK 2022-03-17T19:46:57.2988835Z 2022-03-17T19:46:57.2989133Z Generating XML reports... 2022-03-17T19:46:57.2990780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220317194656.xml 2022-03-17T19:46:59.7682923Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:46:59.7694879Z 2022-03-17T19:46:59.7695175Z Running tests... 2022-03-17T19:46:59.7695710Z ---------------------------------------------------------------------- 2022-03-17T19:47:00.3525343Z test_default_store_timeout_gloo (__main__.TimeoutTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:47:00.3527516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:47:02.3784593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:47:02.3786670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:47:03.4084933Z ok (3.639s) 2022-03-17T19:47:03.4085486Z 2022-03-17T19:47:03.4086599Z ---------------------------------------------------------------------- 2022-03-17T19:47:03.4087602Z Ran 1 test in 3.639s 2022-03-17T19:47:03.4088109Z 2022-03-17T19:47:03.4088362Z OK 2022-03-17T19:47:03.4088690Z 2022-03-17T19:47:03.4088964Z Generating XML reports... 2022-03-17T19:47:03.4150738Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220317194659.xml 2022-03-17T19:47:06.6022863Z Running distributed/test_c10d_nccl ... [2022-03-17 19:47:06.601265] 2022-03-17T19:47:06.6025138Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:47:06.601438] 2022-03-17T19:47:07.4761707Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>]> 2022-03-17T19:47:07.4765938Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-03-17T19:47:07.4766376Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-03-17T19:47:07.4766743Z test_nccl_barrier (__main__.CommTest) 2022-03-17T19:47:07.4767094Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-03-17T19:47:07.4767522Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-03-17T19:47:07.4767916Z test_nccl_barrier_timeout (__main__.CommTest) 2022-03-17T19:47:07.4768305Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-03-17T19:47:07.4768743Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-03-17T19:47:07.4769158Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-03-17T19:47:07.4769572Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-03-17T19:47:07.4769966Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-03-17T19:47:07.4770393Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-03-17T19:47:07.4770826Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-03-17T19:47:07.4771244Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-03-17T19:47:07.4771672Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-03-17T19:47:07.4772048Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-03-17T19:47:07.4781801Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>]> 2022-03-17T19:47:07.4790414Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4790890Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4791336Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4791763Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4792200Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4792616Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4793005Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4793431Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4793849Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4794293Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4794750Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4795190Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4795665Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4796163Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4796623Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4797087Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4797520Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4797989Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4798469Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4798944Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4799427Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4799862Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4800300Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4800817Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4801257Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4801707Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4802283Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4802693Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4803148Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4803625Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4804133Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4804607Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4805137Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4805677Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4806200Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4806758Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4807326Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4807983Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4808479Z test_fp16 (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4808915Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4809410Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4809866Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4810388Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4810928Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4811402Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4811907Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4812428Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4812992Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4813560Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4814070Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4814558Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4815060Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4815603Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4816171Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4816720Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4817245Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4817705Z test_no_grad (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4818069Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4818463Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4818839Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4819261Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:47:07.4819671Z 2022-03-17T19:47:07.4820875Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-03-17T19:47:07.4822099Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4822489Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4822857Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4823241Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4823634Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4824008Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4824395Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4824755Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4825158Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-03-17T19:47:07.4825602Z ]> 2022-03-17T19:47:07.4826018Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-03-17T19:47:07.4827879Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>]> 2022-03-17T19:47:07.4829787Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4830158Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4830498Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4830849Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4831188Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4831510Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4831863Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4832195Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4832544Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4832886Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4833203Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4833556Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4833971Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4834338Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4834669Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4835013Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4835361Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-03-17T19:47:07.4835747Z ]> 2022-03-17T19:47:07.4836142Z test_common_errors (__main__.RendezvousEnvTest) 2022-03-17T19:47:07.4836438Z 2022-03-17T19:47:07.4836831Z ]> 2022-03-17T19:47:07.4837239Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-03-17T19:47:08.3625356Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:08.3637272Z 2022-03-17T19:47:08.3637924Z Running tests... 2022-03-17T19:47:08.3639328Z ---------------------------------------------------------------------- 2022-03-17T19:47:08.9497587Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45806 2022-03-17T19:47:08.9585894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45807 2022-03-17T19:47:09.8451156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:09.8755897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:10.2258176Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:10.2907295Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:12.1666658Z ok (3.802s) 2022-03-17T19:47:12.1667272Z 2022-03-17T19:47:12.1668285Z ---------------------------------------------------------------------- 2022-03-17T19:47:12.1669169Z Ran 1 test in 3.803s 2022-03-17T19:47:12.1669614Z 2022-03-17T19:47:12.1669809Z OK 2022-03-17T19:47:12.1670146Z 2022-03-17T19:47:12.1670459Z Generating XML reports... 2022-03-17T19:47:12.1745016Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194708.xml 2022-03-17T19:47:14.6296649Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:14.6308175Z 2022-03-17T19:47:14.6308470Z Running tests... 2022-03-17T19:47:14.6309032Z ---------------------------------------------------------------------- 2022-03-17T19:47:15.2182566Z test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46021 2022-03-17T19:47:15.2216194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46022 2022-03-17T19:47:16.0774553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:16.1435719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:16.5361676Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:16.7647703Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:18.8304658Z ok (4.199s) 2022-03-17T19:47:18.8305031Z 2022-03-17T19:47:18.8306030Z ---------------------------------------------------------------------- 2022-03-17T19:47:18.8306863Z Ran 1 test in 4.199s 2022-03-17T19:47:18.8307270Z 2022-03-17T19:47:18.8307456Z OK 2022-03-17T19:47:18.8307768Z 2022-03-17T19:47:18.8308079Z Generating XML reports... 2022-03-17T19:47:18.8381005Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194714.xml 2022-03-17T19:47:21.1952090Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:21.1963295Z 2022-03-17T19:47:21.1963880Z Running tests... 2022-03-17T19:47:21.1964979Z ---------------------------------------------------------------------- 2022-03-17T19:47:21.7883431Z test_nccl_barrier (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46236 2022-03-17T19:47:21.7916652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46237 2022-03-17T19:47:22.6963918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:22.7033524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:24.4984521Z skip (3.302s) 2022-03-17T19:47:24.4985012Z 2022-03-17T19:47:24.4985898Z ---------------------------------------------------------------------- 2022-03-17T19:47:24.4986742Z Ran 1 test in 3.302s 2022-03-17T19:47:24.4987090Z 2022-03-17T19:47:24.4987353Z OK (skipped=1) 2022-03-17T19:47:24.4987726Z 2022-03-17T19:47:24.4988024Z Generating XML reports... 2022-03-17T19:47:24.5058797Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194721.xml 2022-03-17T19:47:26.9488238Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:26.9500531Z 2022-03-17T19:47:26.9500780Z Running tests... 2022-03-17T19:47:26.9501317Z ---------------------------------------------------------------------- 2022-03-17T19:47:27.5372566Z test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46440 2022-03-17T19:47:27.5415515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46441 2022-03-17T19:47:28.4280975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:28.4281554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:47:28.4711501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:28.4712899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:47:28.4714529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:47:28.4792017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:47:28.5167767Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:28.5630508Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:30.6496386Z ok (3.699s) 2022-03-17T19:47:30.6496897Z 2022-03-17T19:47:30.6497965Z ---------------------------------------------------------------------- 2022-03-17T19:47:30.6498882Z Ran 1 test in 3.700s 2022-03-17T19:47:30.6499271Z 2022-03-17T19:47:30.6499490Z OK 2022-03-17T19:47:30.6499762Z 2022-03-17T19:47:30.6500068Z Generating XML reports... 2022-03-17T19:47:30.6578816Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194726.xml 2022-03-17T19:47:33.1107801Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:33.1119099Z 2022-03-17T19:47:33.1119879Z Running tests... 2022-03-17T19:47:33.1121097Z ---------------------------------------------------------------------- 2022-03-17T19:47:33.6971524Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46655 2022-03-17T19:47:33.7016007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46656 2022-03-17T19:47:34.5683090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:34.5684526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:47:34.6174174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:34.6175759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:47:34.6177634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:47:34.6199443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:47:36.9089533Z ok (3.797s) 2022-03-17T19:47:36.9090115Z 2022-03-17T19:47:36.9091349Z ---------------------------------------------------------------------- 2022-03-17T19:47:36.9092151Z Ran 1 test in 3.797s 2022-03-17T19:47:36.9092546Z 2022-03-17T19:47:36.9092911Z OK 2022-03-17T19:47:36.9093206Z 2022-03-17T19:47:36.9093563Z Generating XML reports... 2022-03-17T19:47:36.9171399Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194733.xml 2022-03-17T19:47:39.3293579Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:39.3305960Z 2022-03-17T19:47:39.3306319Z Running tests... 2022-03-17T19:47:39.3306899Z ---------------------------------------------------------------------- 2022-03-17T19:47:39.9165493Z test_nccl_barrier_timeout (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46863 2022-03-17T19:47:39.9205210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46864 2022-03-17T19:47:40.8116019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:40.8506725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:42.7282796Z skip (3.397s) 2022-03-17T19:47:42.7283560Z 2022-03-17T19:47:42.7284564Z ---------------------------------------------------------------------- 2022-03-17T19:47:42.7285425Z Ran 1 test in 3.398s 2022-03-17T19:47:42.7285774Z 2022-03-17T19:47:42.7286026Z OK (skipped=1) 2022-03-17T19:47:42.7286425Z 2022-03-17T19:47:42.7286717Z Generating XML reports... 2022-03-17T19:47:42.7364280Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194739.xml 2022-03-17T19:47:45.1667463Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:45.1679824Z 2022-03-17T19:47:45.1680039Z Running tests... 2022-03-17T19:47:45.1680669Z ---------------------------------------------------------------------- 2022-03-17T19:47:45.7539079Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47067 2022-03-17T19:47:45.7570248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47068 2022-03-17T19:47:46.6687598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:46.6751231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:48.4652310Z skip (3.297s) 2022-03-17T19:47:48.4652876Z 2022-03-17T19:47:48.4653915Z ---------------------------------------------------------------------- 2022-03-17T19:47:48.4654877Z Ran 1 test in 3.297s 2022-03-17T19:47:48.4655374Z 2022-03-17T19:47:48.4655724Z OK (skipped=1) 2022-03-17T19:47:48.4656103Z 2022-03-17T19:47:48.4656408Z Generating XML reports... 2022-03-17T19:47:48.4735216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194745.xml 2022-03-17T19:47:51.1251935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:51.1262744Z 2022-03-17T19:47:51.1263075Z Running tests... 2022-03-17T19:47:51.1263614Z ---------------------------------------------------------------------- 2022-03-17T19:47:51.7041510Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip (0.578s) 2022-03-17T19:47:51.7042626Z 2022-03-17T19:47:51.7043871Z ---------------------------------------------------------------------- 2022-03-17T19:47:51.7044696Z Ran 1 test in 0.578s 2022-03-17T19:47:51.7045112Z 2022-03-17T19:47:51.7045369Z OK (skipped=1) 2022-03-17T19:47:51.7045736Z 2022-03-17T19:47:51.7046033Z Generating XML reports... 2022-03-17T19:47:51.7070840Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194751.xml 2022-03-17T19:47:54.1084765Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:47:54.1097258Z 2022-03-17T19:47:54.1097524Z Running tests... 2022-03-17T19:47:54.1098109Z ---------------------------------------------------------------------- 2022-03-17T19:47:54.6967358Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47338 2022-03-17T19:47:54.7006416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47339 2022-03-17T19:47:55.6081937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:55.6299513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:55.6513453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:47:55.6515831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:47:55.6517346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:47:55.6519065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:47:55.6520936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:47:55.6527680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:47:55.6529791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:47:55.6616306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:47:56.0489437Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:56.1912276Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:47:57.8081553Z ok (3.698s) 2022-03-17T19:47:57.8082847Z 2022-03-17T19:47:57.8084209Z ---------------------------------------------------------------------- 2022-03-17T19:47:57.8085222Z Ran 1 test in 3.698s 2022-03-17T19:47:57.8085739Z 2022-03-17T19:47:57.8086008Z OK 2022-03-17T19:47:57.8086401Z 2022-03-17T19:47:57.8086735Z Generating XML reports... 2022-03-17T19:47:57.8164012Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194754.xml 2022-03-17T19:48:00.5283455Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:00.5299294Z 2022-03-17T19:48:00.5299619Z Running tests... 2022-03-17T19:48:00.5300180Z ---------------------------------------------------------------------- 2022-03-17T19:48:01.1172179Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47568 2022-03-17T19:48:01.1221964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47569 2022-03-17T19:48:02.0504792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:02.0506092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:48:02.0727637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:02.0730261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:48:02.0732331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:02.0733639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:48:02.0811433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:02.0812870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:48:02.0814597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:02.0838318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:02.4585432Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:02.6132625Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:04.2298089Z ok (3.700s) 2022-03-17T19:48:04.2298843Z 2022-03-17T19:48:04.2300071Z ---------------------------------------------------------------------- 2022-03-17T19:48:04.2301199Z Ran 1 test in 3.700s 2022-03-17T19:48:04.2301591Z 2022-03-17T19:48:04.2302682Z OK 2022-03-17T19:48:04.2303100Z 2022-03-17T19:48:04.2303519Z Generating XML reports... 2022-03-17T19:48:04.2386289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194800.xml 2022-03-17T19:48:06.6786289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:06.6803153Z 2022-03-17T19:48:06.6803876Z Running tests... 2022-03-17T19:48:06.6805096Z ---------------------------------------------------------------------- 2022-03-17T19:48:07.2654036Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47789 2022-03-17T19:48:07.2699230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47790 2022-03-17T19:48:08.1528497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:08.1529412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:48:08.1705968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:08.1706848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:48:08.1707926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:08.1708621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:48:08.1734667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:08.1735496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:48:08.1737870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:08.1817055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:08.5588305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:08.7108752Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:10.3784495Z ok (3.698s) 2022-03-17T19:48:10.3785248Z 2022-03-17T19:48:10.3787150Z ---------------------------------------------------------------------- 2022-03-17T19:48:10.3788004Z Ran 1 test in 3.698s 2022-03-17T19:48:10.3788437Z 2022-03-17T19:48:10.3788673Z OK 2022-03-17T19:48:10.3789017Z 2022-03-17T19:48:10.3789961Z Generating XML reports... 2022-03-17T19:48:10.3863505Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194806.xml 2022-03-17T19:48:12.8544533Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:12.8559149Z 2022-03-17T19:48:12.8559907Z Running tests... 2022-03-17T19:48:12.8561090Z ---------------------------------------------------------------------- 2022-03-17T19:48:13.4438850Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48010 2022-03-17T19:48:13.4473941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48011 2022-03-17T19:48:14.3658626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:14.3659937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:48:14.3989958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:14.3991216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:48:14.3993142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:14.3994490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:48:14.4072869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:14.4074217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:48:14.4075924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:14.4101211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:14.7923973Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:14.8490819Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:16.5586296Z ok (3.702s) 2022-03-17T19:48:16.5586723Z 2022-03-17T19:48:16.5588247Z ---------------------------------------------------------------------- 2022-03-17T19:48:16.5589339Z Ran 1 test in 3.703s 2022-03-17T19:48:16.5589862Z 2022-03-17T19:48:16.5590098Z OK 2022-03-17T19:48:16.5590536Z 2022-03-17T19:48:16.5590902Z Generating XML reports... 2022-03-17T19:48:16.5668244Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194812.xml 2022-03-17T19:48:19.2706576Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:19.2720728Z 2022-03-17T19:48:19.2721042Z Running tests... 2022-03-17T19:48:19.2721749Z ---------------------------------------------------------------------- 2022-03-17T19:48:19.8594755Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48229 2022-03-17T19:48:19.8630040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48230 2022-03-17T19:48:20.7143929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:20.7145512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:48:20.7470050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:20.7471718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:48:20.7473881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:20.7556356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:20.7664779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:48:20.7666366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:48:20.7668229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:20.7670131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:21.1656623Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:21.2155274Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:22.8720225Z ok (3.599s) 2022-03-17T19:48:22.8720885Z 2022-03-17T19:48:22.8722433Z ---------------------------------------------------------------------- 2022-03-17T19:48:22.8723502Z Ran 1 test in 3.600s 2022-03-17T19:48:22.8724030Z 2022-03-17T19:48:22.8724425Z OK 2022-03-17T19:48:22.8724756Z 2022-03-17T19:48:22.8725061Z Generating XML reports... 2022-03-17T19:48:22.8802602Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194819.xml 2022-03-17T19:48:25.3424535Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:25.3437830Z 2022-03-17T19:48:25.3438040Z Running tests... 2022-03-17T19:48:25.3438497Z ---------------------------------------------------------------------- 2022-03-17T19:48:25.9321195Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48450 2022-03-17T19:48:25.9366629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48451 2022-03-17T19:48:26.8237143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:26.8391942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:28.8444696Z skip (3.500s) 2022-03-17T19:48:28.8445236Z 2022-03-17T19:48:28.8446463Z ---------------------------------------------------------------------- 2022-03-17T19:48:28.8447541Z Ran 1 test in 3.500s 2022-03-17T19:48:28.8447927Z 2022-03-17T19:48:28.8448138Z OK (skipped=1) 2022-03-17T19:48:28.8448504Z 2022-03-17T19:48:28.8448827Z Generating XML reports... 2022-03-17T19:48:28.8528485Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194825.xml 2022-03-17T19:48:31.2862861Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:31.2874807Z 2022-03-17T19:48:31.2874954Z Running tests... 2022-03-17T19:48:31.2875399Z ---------------------------------------------------------------------- 2022-03-17T19:48:31.8762784Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48654 2022-03-17T19:48:31.8802127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48655 2022-03-17T19:48:32.7577915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:32.7581056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:48:32.7960496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:32.7962950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:48:32.7964907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:32.7989892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:33.1784275Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:33.2310008Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:34.8885794Z ok (3.601s) 2022-03-17T19:48:34.8886285Z 2022-03-17T19:48:34.8887366Z ---------------------------------------------------------------------- 2022-03-17T19:48:34.8888171Z Ran 1 test in 3.601s 2022-03-17T19:48:34.8888567Z 2022-03-17T19:48:34.8890191Z OK 2022-03-17T19:48:34.8893601Z 2022-03-17T19:48:34.8893958Z Generating XML reports... 2022-03-17T19:48:34.8968048Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194831.xml 2022-03-17T19:48:37.4672716Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:37.4685525Z 2022-03-17T19:48:37.4685837Z Running tests... 2022-03-17T19:48:37.4686286Z ---------------------------------------------------------------------- 2022-03-17T19:48:38.0565168Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48869 2022-03-17T19:48:38.0600773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48870 2022-03-17T19:48:38.9788640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:38.9789317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:48:38.9875644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:38.9879458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:48:38.9881220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:38.9882767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:48:38.9894730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:48:38.9896116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:48:38.9897890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:38.9990354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:48:39.3877041Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:39.4412055Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:41.4678635Z ok (3.999s) 2022-03-17T19:48:41.4679271Z 2022-03-17T19:48:41.4680293Z ---------------------------------------------------------------------- 2022-03-17T19:48:41.4681158Z Ran 1 test in 3.999s 2022-03-17T19:48:41.4681511Z 2022-03-17T19:48:41.4681722Z OK 2022-03-17T19:48:41.4682224Z 2022-03-17T19:48:41.4682531Z Generating XML reports... 2022-03-17T19:48:41.4760726Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194837.xml 2022-03-17T19:48:43.9061706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:48:43.9073767Z 2022-03-17T19:48:43.9074192Z Running tests... 2022-03-17T19:48:43.9074661Z ---------------------------------------------------------------------- 2022-03-17T19:48:44.4940025Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49088 2022-03-17T19:48:44.4972041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49089 2022-03-17T19:48:45.3993288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:45.4176541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:45.7959094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:45.8563842Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:48:55.4409649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:48:55.4411143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:48:58.0218545Z ok (14.114s) 2022-03-17T19:48:58.0218937Z 2022-03-17T19:48:58.0220387Z ---------------------------------------------------------------------- 2022-03-17T19:48:58.0221266Z Ran 1 test in 14.114s 2022-03-17T19:48:58.0221695Z 2022-03-17T19:48:58.0221873Z OK 2022-03-17T19:48:58.0222205Z 2022-03-17T19:48:58.0222523Z Generating XML reports... 2022-03-17T19:48:58.0300787Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194843.xml 2022-03-17T19:49:00.6483017Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:49:00.6494598Z 2022-03-17T19:49:00.6494750Z Running tests... 2022-03-17T19:49:00.6495350Z ---------------------------------------------------------------------- 2022-03-17T19:49:01.2370288Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49307 2022-03-17T19:49:01.2402707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49308 2022-03-17T19:49:02.1709016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:02.1916368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:02.5737326Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:02.6230256Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:12.1452988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:49:12.1455286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:49:14.6669449Z ok (14.017s) 2022-03-17T19:49:14.6669905Z 2022-03-17T19:49:14.6671162Z ---------------------------------------------------------------------- 2022-03-17T19:49:14.6672334Z Ran 1 test in 14.017s 2022-03-17T19:49:14.6672846Z 2022-03-17T19:49:14.6673021Z OK 2022-03-17T19:49:14.6673636Z 2022-03-17T19:49:14.6673950Z Generating XML reports... 2022-03-17T19:49:14.6753890Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194900.xml 2022-03-17T19:49:17.1308281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:49:17.1320828Z 2022-03-17T19:49:17.1321490Z Running tests... 2022-03-17T19:49:17.1322864Z ---------------------------------------------------------------------- 2022-03-17T19:49:17.7195073Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49526 2022-03-17T19:49:17.7231816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49527 2022-03-17T19:49:18.6492914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:18.6579580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:19.0371725Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:19.0962057Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:28.3509707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:49:28.3511981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:49:31.0437001Z ok (13.911s) 2022-03-17T19:49:31.0437461Z 2022-03-17T19:49:31.0438479Z ---------------------------------------------------------------------- 2022-03-17T19:49:31.0439315Z Ran 1 test in 13.911s 2022-03-17T19:49:31.0439661Z 2022-03-17T19:49:31.0439868Z OK 2022-03-17T19:49:31.0440175Z 2022-03-17T19:49:31.0440468Z Generating XML reports... 2022-03-17T19:49:31.0517934Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194917.xml 2022-03-17T19:49:33.5203756Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:49:33.5216381Z 2022-03-17T19:49:33.5216728Z Running tests... 2022-03-17T19:49:33.5217225Z ---------------------------------------------------------------------- 2022-03-17T19:49:34.1110209Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49745 2022-03-17T19:49:34.1145460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49746 2022-03-17T19:49:35.0130625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:35.0468615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:35.3927109Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:35.4583328Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:44.8342038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:49:44.8369842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:49:47.3451126Z ok (13.823s) 2022-03-17T19:49:47.3451526Z 2022-03-17T19:49:47.3452525Z ---------------------------------------------------------------------- 2022-03-17T19:49:47.3453279Z Ran 1 test in 13.823s 2022-03-17T19:49:47.3453665Z 2022-03-17T19:49:47.3453840Z OK 2022-03-17T19:49:47.3454162Z 2022-03-17T19:49:47.3454475Z Generating XML reports... 2022-03-17T19:49:47.3533672Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194933.xml 2022-03-17T19:49:49.7810261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:49:49.7819901Z 2022-03-17T19:49:49.7820520Z Running tests... 2022-03-17T19:49:49.7821570Z ---------------------------------------------------------------------- 2022-03-17T19:49:49.7824942Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip (0.001s) 2022-03-17T19:49:49.7825785Z 2022-03-17T19:49:49.7826540Z ---------------------------------------------------------------------- 2022-03-17T19:49:49.7827358Z Ran 1 test in 0.001s 2022-03-17T19:49:49.7827750Z 2022-03-17T19:49:49.7828027Z OK (skipped=1) 2022-03-17T19:49:49.7828407Z 2022-03-17T19:49:49.7828670Z Generating XML reports... 2022-03-17T19:49:49.7856948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194949.xml 2022-03-17T19:49:52.2405680Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:49:52.2418810Z 2022-03-17T19:49:52.2419083Z Running tests... 2022-03-17T19:49:52.2419578Z ---------------------------------------------------------------------- 2022-03-17T19:49:52.2421282Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip (0.001s) 2022-03-17T19:49:52.2421614Z 2022-03-17T19:49:52.2422120Z ---------------------------------------------------------------------- 2022-03-17T19:49:52.2422479Z Ran 1 test in 0.001s 2022-03-17T19:49:52.2422641Z 2022-03-17T19:49:52.2422777Z OK (skipped=1) 2022-03-17T19:49:52.2422950Z 2022-03-17T19:49:52.2423064Z Generating XML reports... 2022-03-17T19:49:52.2453865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194952.xml 2022-03-17T19:49:54.5792391Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:49:54.5806658Z 2022-03-17T19:49:54.5807000Z Running tests... 2022-03-17T19:49:54.5807441Z ---------------------------------------------------------------------- 2022-03-17T19:49:55.1690935Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50098 2022-03-17T19:49:55.1735566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50099 2022-03-17T19:49:56.0934365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:56.1204900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:56.5068176Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:56.5574518Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:49:58.6825781Z ok (4.102s) 2022-03-17T19:49:58.6826225Z 2022-03-17T19:49:58.6827184Z ---------------------------------------------------------------------- 2022-03-17T19:49:58.6827991Z Ran 1 test in 4.102s 2022-03-17T19:49:58.6832179Z 2022-03-17T19:49:58.6832900Z OK 2022-03-17T19:49:58.6833282Z 2022-03-17T19:49:58.6833600Z Generating XML reports... 2022-03-17T19:49:58.6893195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194954.xml 2022-03-17T19:50:01.1125609Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:50:01.1140130Z 2022-03-17T19:50:01.1140675Z Running tests... 2022-03-17T19:50:01.1141136Z ---------------------------------------------------------------------- 2022-03-17T19:50:01.7026097Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50317 2022-03-17T19:50:01.7058813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50318 2022-03-17T19:50:02.5576730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:02.5743926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:02.9545332Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:03.0064248Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:05.3149325Z ok (4.200s) 2022-03-17T19:50:05.3149785Z 2022-03-17T19:50:05.3150751Z ---------------------------------------------------------------------- 2022-03-17T19:50:05.3151588Z Ran 1 test in 4.201s 2022-03-17T19:50:05.3151986Z 2022-03-17T19:50:05.3152153Z OK 2022-03-17T19:50:05.3152468Z 2022-03-17T19:50:05.3159513Z Generating XML reports... 2022-03-17T19:50:05.3230123Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195001.xml 2022-03-17T19:50:07.7142164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:50:07.7148515Z 2022-03-17T19:50:07.7148774Z Running tests... 2022-03-17T19:50:07.7150047Z ---------------------------------------------------------------------- 2022-03-17T19:50:07.7157146Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:50:08.3042651Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50536 2022-03-17T19:50:08.3078770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50537 2022-03-17T19:50:09.2431966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:09.2461940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:09.6297493Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:09.6916736Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:21.8327162Z ok (14.117s) 2022-03-17T19:50:21.8327629Z 2022-03-17T19:50:21.8328573Z ---------------------------------------------------------------------- 2022-03-17T19:50:21.8329435Z Ran 1 test in 14.117s 2022-03-17T19:50:21.8329821Z 2022-03-17T19:50:21.8329988Z OK 2022-03-17T19:50:21.8330290Z 2022-03-17T19:50:21.8330595Z Generating XML reports... 2022-03-17T19:50:21.8409420Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195007.xml 2022-03-17T19:50:24.2581014Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:50:24.2593202Z 2022-03-17T19:50:24.2593736Z Running tests... 2022-03-17T19:50:24.2595256Z ---------------------------------------------------------------------- 2022-03-17T19:50:24.2602534Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:50:24.8498804Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50755 2022-03-17T19:50:24.8538797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50756 2022-03-17T19:50:25.7398520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:25.7713306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:26.1610709Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:26.2265371Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:38.0787771Z ok (13.819s) 2022-03-17T19:50:38.0788377Z 2022-03-17T19:50:38.0789559Z ---------------------------------------------------------------------- 2022-03-17T19:50:38.0790461Z Ran 1 test in 13.819s 2022-03-17T19:50:38.0790874Z 2022-03-17T19:50:38.0791060Z OK 2022-03-17T19:50:38.0791402Z 2022-03-17T19:50:38.0792049Z Generating XML reports... 2022-03-17T19:50:38.0865571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195024.xml 2022-03-17T19:50:40.4349084Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:50:40.4360582Z 2022-03-17T19:50:40.4361151Z Running tests... 2022-03-17T19:50:40.4362564Z ---------------------------------------------------------------------- 2022-03-17T19:50:40.4371874Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:50:41.0261629Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50974 2022-03-17T19:50:41.0319485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50975 2022-03-17T19:50:41.9457471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:41.9667786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:42.3252091Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:42.3804524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:51.6382927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.7208948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.7605528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.7631620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.7840469Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:50:51.7843245Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:50:51.7852127Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:50:51.7855234Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:50:51.7997372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.7998945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.8262015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.8274400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.8622666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.8647816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.8983258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:51.8996275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:50:54.4633222Z ok (14.027s) 2022-03-17T19:50:54.4634571Z 2022-03-17T19:50:54.4635769Z ---------------------------------------------------------------------- 2022-03-17T19:50:54.4636800Z Ran 1 test in 14.027s 2022-03-17T19:50:54.4637148Z 2022-03-17T19:50:54.4637400Z OK 2022-03-17T19:50:54.4637819Z 2022-03-17T19:50:54.4638226Z Generating XML reports... 2022-03-17T19:50:54.4716908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195040.xml 2022-03-17T19:50:56.9263489Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:50:56.9273850Z 2022-03-17T19:50:56.9274799Z Running tests... 2022-03-17T19:50:56.9276312Z ---------------------------------------------------------------------- 2022-03-17T19:50:56.9283593Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:50:57.5159692Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51193 2022-03-17T19:50:57.5195480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51194 2022-03-17T19:50:58.3993295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:58.4242502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:58.8224385Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:50:58.8759356Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:08.2176945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.3037631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.3468569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.3471229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.3683811Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:51:08.3686119Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:51:08.3689428Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:51:08.3691791Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:51:08.3840340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.3844600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.4137431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.4139308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.4538839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.4540379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.4863509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:08.4875172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:11.1460422Z ok (14.218s) 2022-03-17T19:51:11.1460952Z 2022-03-17T19:51:11.1462196Z ---------------------------------------------------------------------- 2022-03-17T19:51:11.1463297Z Ran 1 test in 14.219s 2022-03-17T19:51:11.1463796Z 2022-03-17T19:51:11.1464612Z OK 2022-03-17T19:51:11.1465000Z 2022-03-17T19:51:11.1465413Z Generating XML reports... 2022-03-17T19:51:11.1542224Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195056.xml 2022-03-17T19:51:13.5720878Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:51:13.5736451Z 2022-03-17T19:51:13.5736603Z Running tests... 2022-03-17T19:51:13.5737118Z ---------------------------------------------------------------------- 2022-03-17T19:51:13.5741351Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:14.1612883Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51412 2022-03-17T19:51:14.1646670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51413 2022-03-17T19:51:15.0461665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:15.0633011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:15.4239402Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:15.4775209Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:24.8482715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:24.8484027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:24.8909382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:24.8911376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:27.4954725Z ok (13.921s) 2022-03-17T19:51:27.4955299Z 2022-03-17T19:51:27.4956666Z ---------------------------------------------------------------------- 2022-03-17T19:51:27.4957696Z Ran 1 test in 13.922s 2022-03-17T19:51:27.4958164Z 2022-03-17T19:51:27.4958453Z OK 2022-03-17T19:51:27.4958822Z 2022-03-17T19:51:27.4959167Z Generating XML reports... 2022-03-17T19:51:27.5035908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195113.xml 2022-03-17T19:51:29.9509465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:51:29.9520096Z 2022-03-17T19:51:29.9520388Z Running tests... 2022-03-17T19:51:29.9520905Z ---------------------------------------------------------------------- 2022-03-17T19:51:29.9528371Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:51:30.5399033Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51631 2022-03-17T19:51:30.5442582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51632 2022-03-17T19:51:31.4025567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:31.4026868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:31.7866065Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:31.8386056Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:41.8409152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:41.8410804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:41.8853065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:41.8854412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:44.7746312Z ok (14.822s) 2022-03-17T19:51:44.7746866Z 2022-03-17T19:51:44.7748211Z ---------------------------------------------------------------------- 2022-03-17T19:51:44.7749326Z Ran 1 test in 14.822s 2022-03-17T19:51:44.7749790Z 2022-03-17T19:51:44.7750048Z OK 2022-03-17T19:51:44.7750444Z 2022-03-17T19:51:44.7751020Z Generating XML reports... 2022-03-17T19:51:44.7827909Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195129.xml 2022-03-17T19:51:47.2558256Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:51:47.2568739Z 2022-03-17T19:51:47.2569076Z Running tests... 2022-03-17T19:51:47.2569501Z ---------------------------------------------------------------------- 2022-03-17T19:51:47.2581946Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:47.8462345Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51850 2022-03-17T19:51:47.8503192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51851 2022-03-17T19:51:48.6997025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:48.7351980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:49.0800062Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:49.1404460Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:51:58.3390962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:58.4051524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:58.4365336Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:51:58.4369700Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:51:58.4840448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:51:58.4841772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:01.0788007Z ok (13.821s) 2022-03-17T19:52:01.0788417Z 2022-03-17T19:52:01.0789391Z ---------------------------------------------------------------------- 2022-03-17T19:52:01.0790842Z Ran 1 test in 13.822s 2022-03-17T19:52:01.0791292Z 2022-03-17T19:52:01.0791524Z OK 2022-03-17T19:52:01.0791856Z 2022-03-17T19:52:01.0800168Z Generating XML reports... 2022-03-17T19:52:01.0869999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195147.xml 2022-03-17T19:52:03.5212670Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:52:03.5224499Z 2022-03-17T19:52:03.5225133Z Running tests... 2022-03-17T19:52:03.5226347Z ---------------------------------------------------------------------- 2022-03-17T19:52:03.5239966Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:52:04.1101313Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52069 2022-03-17T19:52:04.1140557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52070 2022-03-17T19:52:05.0002750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:05.0240361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:05.4128756Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:05.4630603Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:17.4391266Z ok (13.916s) 2022-03-17T19:52:17.4391668Z 2022-03-17T19:52:17.4392796Z ---------------------------------------------------------------------- 2022-03-17T19:52:17.4393771Z Ran 1 test in 13.916s 2022-03-17T19:52:17.4394211Z 2022-03-17T19:52:17.4394462Z OK 2022-03-17T19:52:17.4394794Z 2022-03-17T19:52:17.4395140Z Generating XML reports... 2022-03-17T19:52:17.4472555Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195203.xml 2022-03-17T19:52:19.9721700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:52:19.9729489Z 2022-03-17T19:52:19.9729904Z Running tests... 2022-03-17T19:52:19.9731125Z ---------------------------------------------------------------------- 2022-03-17T19:52:19.9741378Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:52:20.5607635Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52288 2022-03-17T19:52:20.5647299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52289 2022-03-17T19:52:21.4544008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:21.4660635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:21.8417084Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:21.9007617Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:31.2906316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:31.2917324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:31.3343547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:31.3347501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:33.9892587Z ok (14.016s) 2022-03-17T19:52:33.9893062Z 2022-03-17T19:52:33.9894211Z ---------------------------------------------------------------------- 2022-03-17T19:52:33.9895173Z Ran 1 test in 14.016s 2022-03-17T19:52:33.9895618Z 2022-03-17T19:52:33.9895859Z OK 2022-03-17T19:52:33.9896220Z 2022-03-17T19:52:33.9901503Z Generating XML reports... 2022-03-17T19:52:33.9974305Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195219.xml 2022-03-17T19:52:36.7001726Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:52:36.7013206Z 2022-03-17T19:52:36.7013463Z Running tests... 2022-03-17T19:52:36.7013904Z ---------------------------------------------------------------------- 2022-03-17T19:52:36.7026309Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:52:37.2920036Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52507 2022-03-17T19:52:37.2965021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52508 2022-03-17T19:52:38.1577518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:38.2381502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:38.5428583Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:38.6226229Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:47.7283830Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:52:47.7569609Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:52:47.9239375Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:52:47.9241785Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:52:47.9245689Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:52:47.9248036Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:52:47.9369903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.9373406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:48.0074195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:48.0092555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:50.5207062Z ok (13.819s) 2022-03-17T19:52:50.5207453Z 2022-03-17T19:52:50.5208377Z ---------------------------------------------------------------------- 2022-03-17T19:52:50.5209195Z Ran 1 test in 13.819s 2022-03-17T19:52:50.5210056Z 2022-03-17T19:52:50.5210250Z OK 2022-03-17T19:52:50.5210535Z 2022-03-17T19:52:50.5210824Z Generating XML reports... 2022-03-17T19:52:50.5291519Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195236.xml 2022-03-17T19:52:52.9685711Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:52:52.9696338Z 2022-03-17T19:52:52.9697061Z Running tests... 2022-03-17T19:52:52.9698260Z ---------------------------------------------------------------------- 2022-03-17T19:52:52.9707330Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:52:53.5581416Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52726 2022-03-17T19:52:53.5613844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52727 2022-03-17T19:52:54.4800672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:54.4904257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:54.8756651Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:52:54.9295014Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:04.1242949Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:53:04.1245276Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:53:04.1265069Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:53:04.1267261Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:53:04.2319300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:04.2320737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:04.2897158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:04.2900422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:06.7923924Z ok (13.822s) 2022-03-17T19:53:06.7924315Z 2022-03-17T19:53:06.7925149Z ---------------------------------------------------------------------- 2022-03-17T19:53:06.7925957Z Ran 1 test in 13.823s 2022-03-17T19:53:06.7926458Z 2022-03-17T19:53:06.7926658Z OK 2022-03-17T19:53:06.7926953Z 2022-03-17T19:53:06.7927272Z Generating XML reports... 2022-03-17T19:53:06.8010612Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195252.xml 2022-03-17T19:53:09.2772521Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:53:09.2784191Z 2022-03-17T19:53:09.2784944Z Running tests... 2022-03-17T19:53:09.2799239Z ---------------------------------------------------------------------- 2022-03-17T19:53:09.2800478Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:53:09.8679111Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52945 2022-03-17T19:53:09.8714744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52946 2022-03-17T19:53:10.7970519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:10.8033192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:11.1805146Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:11.2405788Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:20.9108843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:20.9742463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:21.0291588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:21.0292158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:23.4973418Z ok (14.219s) 2022-03-17T19:53:23.4973621Z 2022-03-17T19:53:23.4975072Z ---------------------------------------------------------------------- 2022-03-17T19:53:23.4975847Z Ran 1 test in 14.219s 2022-03-17T19:53:23.4976209Z 2022-03-17T19:53:23.4976374Z OK 2022-03-17T19:53:23.4976660Z 2022-03-17T19:53:23.4976935Z Generating XML reports... 2022-03-17T19:53:23.5057755Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195309.xml 2022-03-17T19:53:26.2963936Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:53:26.2976738Z 2022-03-17T19:53:26.2976989Z Running tests... 2022-03-17T19:53:26.2977428Z ---------------------------------------------------------------------- 2022-03-17T19:53:26.2989825Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:53:26.8863501Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53164 2022-03-17T19:53:26.8905296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53165 2022-03-17T19:53:27.7409568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:27.8338733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:28.2177332Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:28.2730089Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:37.6915572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.6918379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.7327874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.7329114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.7603923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.7610222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.8014726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.8015985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:40.4155242Z ok (14.118s) 2022-03-17T19:53:40.4155640Z 2022-03-17T19:53:40.4156588Z ---------------------------------------------------------------------- 2022-03-17T19:53:40.4157347Z Ran 1 test in 14.118s 2022-03-17T19:53:40.4157712Z 2022-03-17T19:53:40.4157912Z OK 2022-03-17T19:53:40.4167853Z 2022-03-17T19:53:40.4169272Z Generating XML reports... 2022-03-17T19:53:40.4238611Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195326.xml 2022-03-17T19:53:42.9247715Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:53:42.9257319Z 2022-03-17T19:53:42.9257594Z Running tests... 2022-03-17T19:53:42.9259799Z ---------------------------------------------------------------------- 2022-03-17T19:53:43.5160462Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53383 2022-03-17T19:53:43.5199646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53384 2022-03-17T19:53:44.4568502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:44.4993488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:44.8357540Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:44.8881208Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:47.4290495Z ok (4.503s) 2022-03-17T19:53:47.4290896Z 2022-03-17T19:53:47.4292246Z ---------------------------------------------------------------------- 2022-03-17T19:53:47.4293108Z Ran 1 test in 4.503s 2022-03-17T19:53:47.4293555Z 2022-03-17T19:53:47.4293761Z OK 2022-03-17T19:53:47.4294079Z 2022-03-17T19:53:47.4294347Z Generating XML reports... 2022-03-17T19:53:47.4362960Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195342.xml 2022-03-17T19:53:50.1952452Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:53:50.1964943Z 2022-03-17T19:53:50.1965429Z Running tests... 2022-03-17T19:53:50.1966410Z ---------------------------------------------------------------------- 2022-03-17T19:53:50.7907226Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53602 2022-03-17T19:53:50.7945790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53603 2022-03-17T19:53:51.7171235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:51.7175619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:52.1046945Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:52.1595757Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:54.5094286Z ok (4.312s) 2022-03-17T19:53:54.5094691Z 2022-03-17T19:53:54.5095903Z ---------------------------------------------------------------------- 2022-03-17T19:53:54.5096935Z Ran 1 test in 4.313s 2022-03-17T19:53:54.5097414Z 2022-03-17T19:53:54.5097662Z OK 2022-03-17T19:53:54.5098019Z 2022-03-17T19:53:54.5098306Z Generating XML reports... 2022-03-17T19:53:54.5175160Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195350.xml 2022-03-17T19:53:56.9424535Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:53:56.9436432Z 2022-03-17T19:53:56.9437064Z Running tests... 2022-03-17T19:53:56.9438216Z ---------------------------------------------------------------------- 2022-03-17T19:53:57.5528796Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53821 2022-03-17T19:53:57.5580625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53822 2022-03-17T19:53:58.4341330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:58.4479192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:58.8117159Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:53:58.8678268Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:00.9669894Z ok (4.023s) 2022-03-17T19:54:00.9670303Z 2022-03-17T19:54:00.9671509Z ---------------------------------------------------------------------- 2022-03-17T19:54:00.9672528Z Ran 1 test in 4.023s 2022-03-17T19:54:00.9672899Z 2022-03-17T19:54:00.9673082Z OK 2022-03-17T19:54:00.9673359Z 2022-03-17T19:54:00.9673637Z Generating XML reports... 2022-03-17T19:54:00.9747142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195356.xml 2022-03-17T19:54:03.5176305Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:03.5190864Z 2022-03-17T19:54:03.5191128Z Running tests... 2022-03-17T19:54:03.5192068Z ---------------------------------------------------------------------- 2022-03-17T19:54:03.5209025Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:54:04.1254706Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54040 2022-03-17T19:54:04.1308216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54041 2022-03-17T19:54:05.0628296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:05.0671207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:05.4462224Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:05.5081652Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:07.2391606Z ok (3.720s) 2022-03-17T19:54:07.2392020Z 2022-03-17T19:54:07.2392956Z ---------------------------------------------------------------------- 2022-03-17T19:54:07.2393734Z Ran 1 test in 3.720s 2022-03-17T19:54:07.2394121Z 2022-03-17T19:54:07.2394309Z OK 2022-03-17T19:54:07.2394600Z 2022-03-17T19:54:07.2394879Z Generating XML reports... 2022-03-17T19:54:07.2475435Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195403.xml 2022-03-17T19:54:09.7221565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:09.7236140Z 2022-03-17T19:54:09.7236384Z Running tests... 2022-03-17T19:54:09.7237070Z ---------------------------------------------------------------------- 2022-03-17T19:54:09.7244051Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:54:10.3144545Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54259 2022-03-17T19:54:10.3252823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54260 2022-03-17T19:54:11.2474032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:11.2705211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:11.6588665Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:11.7074448Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:13.8354398Z ok (4.111s) 2022-03-17T19:54:13.8354814Z 2022-03-17T19:54:13.8356138Z ---------------------------------------------------------------------- 2022-03-17T19:54:13.8356917Z Ran 1 test in 4.112s 2022-03-17T19:54:13.8357285Z 2022-03-17T19:54:13.8357489Z OK 2022-03-17T19:54:13.8357782Z 2022-03-17T19:54:13.8358054Z Generating XML reports... 2022-03-17T19:54:13.8441591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195409.xml 2022-03-17T19:54:16.3563752Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:16.3575233Z 2022-03-17T19:54:16.3575426Z Running tests... 2022-03-17T19:54:16.3575859Z ---------------------------------------------------------------------- 2022-03-17T19:54:16.9468246Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54478 2022-03-17T19:54:16.9503770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54479 2022-03-17T19:54:17.8572880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:17.8899259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:20.1588320Z skip (3.801s) 2022-03-17T19:54:20.1588811Z 2022-03-17T19:54:20.1589966Z ---------------------------------------------------------------------- 2022-03-17T19:54:20.1590956Z Ran 1 test in 3.801s 2022-03-17T19:54:20.1591397Z 2022-03-17T19:54:20.1591670Z OK (skipped=1) 2022-03-17T19:54:20.1592010Z 2022-03-17T19:54:20.1592280Z Generating XML reports... 2022-03-17T19:54:20.1672919Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195416.xml 2022-03-17T19:54:22.6029593Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:22.6041526Z 2022-03-17T19:54:22.6042965Z Running tests... 2022-03-17T19:54:22.6044477Z ---------------------------------------------------------------------- 2022-03-17T19:54:23.1965653Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54682 2022-03-17T19:54:23.2069246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54683 2022-03-17T19:54:24.1091589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:24.1449790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:24.5071791Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:24.5484398Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:24.7781042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.7799750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.7991244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.7992785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.8435100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.8436943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.8616596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:24.8618249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:26.5150459Z ok (3.910s) 2022-03-17T19:54:26.5151127Z 2022-03-17T19:54:26.5152463Z ---------------------------------------------------------------------- 2022-03-17T19:54:26.5153276Z Ran 1 test in 3.910s 2022-03-17T19:54:26.5153635Z 2022-03-17T19:54:26.5153827Z OK 2022-03-17T19:54:26.5154165Z 2022-03-17T19:54:26.5156126Z Generating XML reports... 2022-03-17T19:54:26.5222656Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195422.xml 2022-03-17T19:54:29.1117989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:29.1128394Z 2022-03-17T19:54:29.1129257Z Running tests... 2022-03-17T19:54:29.1130441Z ---------------------------------------------------------------------- 2022-03-17T19:54:29.7017103Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54901 2022-03-17T19:54:29.7059276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54902 2022-03-17T19:54:30.6173670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:30.6178573Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-03-17T19:54:30.6179238Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-03-17T19:54:30.6548162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:30.6552653Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-03-17T19:54:30.6553958Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-03-17T19:54:32.2128269Z ok (3.099s) 2022-03-17T19:54:32.2128682Z 2022-03-17T19:54:32.2129775Z ---------------------------------------------------------------------- 2022-03-17T19:54:32.2130789Z Ran 1 test in 3.100s 2022-03-17T19:54:32.2131249Z 2022-03-17T19:54:32.2131497Z OK 2022-03-17T19:54:32.2131821Z 2022-03-17T19:54:32.2132115Z Generating XML reports... 2022-03-17T19:54:32.2207096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195429.xml 2022-03-17T19:54:34.5774439Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:34.5789587Z 2022-03-17T19:54:34.5789920Z Running tests... 2022-03-17T19:54:34.5790516Z ---------------------------------------------------------------------- 2022-03-17T19:54:35.1740270Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55109 2022-03-17T19:54:35.1849008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55110 2022-03-17T19:54:36.1032723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:36.1269367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:36.4835751Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:36.5458620Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:38.3929951Z ok (3.814s) 2022-03-17T19:54:38.3930516Z 2022-03-17T19:54:38.3931706Z ---------------------------------------------------------------------- 2022-03-17T19:54:38.3932685Z Ran 1 test in 3.814s 2022-03-17T19:54:38.3933144Z 2022-03-17T19:54:38.3933327Z OK 2022-03-17T19:54:38.3933624Z 2022-03-17T19:54:38.3933948Z Generating XML reports... 2022-03-17T19:54:38.4015433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195434.xml 2022-03-17T19:54:40.9619341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:40.9631651Z 2022-03-17T19:54:40.9632491Z Running tests... 2022-03-17T19:54:40.9634014Z ---------------------------------------------------------------------- 2022-03-17T19:54:41.5560381Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55328 2022-03-17T19:54:41.5596005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55329 2022-03-17T19:54:42.4973324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:42.4999039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:42.8863117Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:42.9443104Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:44.7669598Z ok (3.803s) 2022-03-17T19:54:44.7670356Z 2022-03-17T19:54:44.7672017Z ---------------------------------------------------------------------- 2022-03-17T19:54:44.7673018Z Ran 1 test in 3.804s 2022-03-17T19:54:44.7673370Z 2022-03-17T19:54:44.7673551Z OK 2022-03-17T19:54:44.7673883Z 2022-03-17T19:54:44.7674160Z Generating XML reports... 2022-03-17T19:54:44.7705154Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195440.xml 2022-03-17T19:54:47.2606882Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:54:47.2620405Z 2022-03-17T19:54:47.2620775Z Running tests... 2022-03-17T19:54:47.2621472Z ---------------------------------------------------------------------- 2022-03-17T19:54:47.8620839Z test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55547 2022-03-17T19:54:47.8653597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55548 2022-03-17T19:54:48.7217017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:48.8091431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:49.1983837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:49.2532070Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:54:58.5854812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:58.5865465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:58.6627484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:58.6628818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:55:01.2926022Z ok (14.030s) 2022-03-17T19:55:01.2926478Z 2022-03-17T19:55:01.2927542Z ---------------------------------------------------------------------- 2022-03-17T19:55:01.2928319Z Ran 1 test in 14.031s 2022-03-17T19:55:01.2928687Z 2022-03-17T19:55:01.2928880Z OK 2022-03-17T19:55:01.2929200Z 2022-03-17T19:55:01.2930104Z Generating XML reports... 2022-03-17T19:55:01.3011590Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195447.xml 2022-03-17T19:55:04.0341195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:55:04.0351901Z 2022-03-17T19:55:04.0352815Z Running tests... 2022-03-17T19:55:04.0353863Z ---------------------------------------------------------------------- 2022-03-17T19:55:04.6268900Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55777 2022-03-17T19:55:04.6305822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55778 2022-03-17T19:55:05.5187313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:05.5411930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:05.5606230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:55:05.5606937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:55:05.5607984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:05.5608784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:05.9505182Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:06.0137345Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:18.0572880Z ok (14.021s) 2022-03-17T19:55:18.0573282Z 2022-03-17T19:55:18.0574461Z ---------------------------------------------------------------------- 2022-03-17T19:55:18.0575461Z Ran 1 test in 14.021s 2022-03-17T19:55:18.0575938Z 2022-03-17T19:55:18.0576213Z OK 2022-03-17T19:55:18.0576598Z 2022-03-17T19:55:18.0576879Z Generating XML reports... 2022-03-17T19:55:18.0644415Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195504.xml 2022-03-17T19:55:20.6583334Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:55:20.6591999Z 2022-03-17T19:55:20.6592634Z Running tests... 2022-03-17T19:55:20.6594071Z ---------------------------------------------------------------------- 2022-03-17T19:55:21.2512067Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56002 2022-03-17T19:55:21.2545226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56003 2022-03-17T19:55:22.1935345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:22.1937596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:55:22.2130854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:22.2137157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:55:22.2138149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:22.2145167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:22.5996043Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:22.6586210Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:34.5825853Z ok (13.923s) 2022-03-17T19:55:34.5826232Z 2022-03-17T19:55:34.5829371Z ---------------------------------------------------------------------- 2022-03-17T19:55:34.5830320Z Ran 1 test in 13.923s 2022-03-17T19:55:34.5830692Z 2022-03-17T19:55:34.5830881Z OK 2022-03-17T19:55:34.5831181Z 2022-03-17T19:55:34.5831450Z Generating XML reports... 2022-03-17T19:55:34.5905404Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195520.xml 2022-03-17T19:55:36.9740101Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:55:36.9754559Z 2022-03-17T19:55:36.9755396Z Running tests... 2022-03-17T19:55:36.9756900Z ---------------------------------------------------------------------- 2022-03-17T19:55:37.5673780Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56221 2022-03-17T19:55:37.5715583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56222 2022-03-17T19:55:38.4497858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:38.4499222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:55:38.4854496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:38.4856059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:55:38.4858146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:38.4908622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:38.8701927Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:38.9225842Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:50.8968177Z ok (13.921s) 2022-03-17T19:55:50.8968636Z 2022-03-17T19:55:50.8969691Z ---------------------------------------------------------------------- 2022-03-17T19:55:50.8970472Z Ran 1 test in 13.921s 2022-03-17T19:55:50.8970835Z 2022-03-17T19:55:50.8976709Z OK 2022-03-17T19:55:50.8977274Z 2022-03-17T19:55:50.8977775Z Generating XML reports... 2022-03-17T19:55:50.9049833Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195536.xml 2022-03-17T19:55:53.8344530Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:55:53.8356851Z 2022-03-17T19:55:53.8356980Z Running tests... 2022-03-17T19:55:53.8357432Z ---------------------------------------------------------------------- 2022-03-17T19:55:54.4317162Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56440 2022-03-17T19:55:54.4358882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56441 2022-03-17T19:55:55.3450293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:55.3551330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:55.3765445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:55:55.3766687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:55:55.3768637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:55.3869360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:55.7738563Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:55:55.8371320Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:07.7645548Z ok (13.928s) 2022-03-17T19:56:07.7646155Z 2022-03-17T19:56:07.7647240Z ---------------------------------------------------------------------- 2022-03-17T19:56:07.7648014Z Ran 1 test in 13.928s 2022-03-17T19:56:07.7648376Z 2022-03-17T19:56:07.7648543Z OK 2022-03-17T19:56:07.7648835Z 2022-03-17T19:56:07.7651795Z Generating XML reports... 2022-03-17T19:56:07.7727022Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195553.xml 2022-03-17T19:56:10.2483432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:56:10.2496605Z 2022-03-17T19:56:10.2496843Z Running tests... 2022-03-17T19:56:10.2497390Z ---------------------------------------------------------------------- 2022-03-17T19:56:10.8429904Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56665 2022-03-17T19:56:10.8463387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56666 2022-03-17T19:56:11.7358953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:11.7371643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:56:11.7655435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:11.7659918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:56:11.7661521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:56:11.7672006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:56:12.1534330Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:12.2084725Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:24.2716622Z ok (14.021s) 2022-03-17T19:56:24.2716985Z 2022-03-17T19:56:24.2717918Z ---------------------------------------------------------------------- 2022-03-17T19:56:24.2718709Z Ran 1 test in 14.022s 2022-03-17T19:56:24.2719071Z 2022-03-17T19:56:24.2719247Z OK 2022-03-17T19:56:24.2719550Z 2022-03-17T19:56:24.2719836Z Generating XML reports... 2022-03-17T19:56:24.2800804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195610.xml 2022-03-17T19:56:26.7657501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:56:26.7670283Z 2022-03-17T19:56:26.7670522Z Running tests... 2022-03-17T19:56:26.7670999Z ---------------------------------------------------------------------- 2022-03-17T19:56:27.3585834Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56884 2022-03-17T19:56:27.3621877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56885 2022-03-17T19:56:28.2595039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:28.2596523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:56:28.2951029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:28.2955052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:56:28.2957052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:56:28.3006863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:56:28.6778513Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:28.7292460Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:40.7953274Z ok (14.028s) 2022-03-17T19:56:40.7953698Z 2022-03-17T19:56:40.7954642Z ---------------------------------------------------------------------- 2022-03-17T19:56:40.7955429Z Ran 1 test in 14.028s 2022-03-17T19:56:40.7955779Z 2022-03-17T19:56:40.7956935Z OK 2022-03-17T19:56:40.7959029Z 2022-03-17T19:56:40.7959883Z Generating XML reports... 2022-03-17T19:56:40.8033279Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195626.xml 2022-03-17T19:56:43.4073326Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:56:43.4086963Z 2022-03-17T19:56:43.4087682Z Running tests... 2022-03-17T19:56:43.4088984Z ---------------------------------------------------------------------- 2022-03-17T19:56:44.0008782Z test_fp16 (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57103 2022-03-17T19:56:44.0047545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57104 2022-03-17T19:56:44.9183727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:44.9524027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:45.3446527Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:45.3976061Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:56:57.3302287Z ok (13.921s) 2022-03-17T19:56:57.3302678Z 2022-03-17T19:56:57.3303640Z ---------------------------------------------------------------------- 2022-03-17T19:56:57.3304429Z Ran 1 test in 13.922s 2022-03-17T19:56:57.3304802Z 2022-03-17T19:56:57.3305003Z OK 2022-03-17T19:56:57.3305301Z 2022-03-17T19:56:57.3305581Z Generating XML reports... 2022-03-17T19:56:57.3382183Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195643.xml 2022-03-17T19:56:59.8196935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:56:59.8210878Z 2022-03-17T19:56:59.8211152Z Running tests... 2022-03-17T19:56:59.8211689Z ---------------------------------------------------------------------- 2022-03-17T19:57:00.4140474Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57322 2022-03-17T19:57:00.4182166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57323 2022-03-17T19:57:01.2803162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:01.2806579Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:01.3284240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:01.3285296Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:01.7103667Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:01.7636545Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:03.9285180Z ok (4.107s) 2022-03-17T19:57:03.9285589Z 2022-03-17T19:57:03.9287232Z ---------------------------------------------------------------------- 2022-03-17T19:57:03.9288052Z Ran 1 test in 4.107s 2022-03-17T19:57:03.9288419Z 2022-03-17T19:57:03.9288617Z OK 2022-03-17T19:57:03.9292535Z 2022-03-17T19:57:03.9293137Z Generating XML reports... 2022-03-17T19:57:03.9367235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195659.xml 2022-03-17T19:57:06.4499231Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:57:06.4512221Z 2022-03-17T19:57:06.4512475Z Running tests... 2022-03-17T19:57:06.4513017Z ---------------------------------------------------------------------- 2022-03-17T19:57:07.0445413Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57541 2022-03-17T19:57:07.0485147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57542 2022-03-17T19:57:07.9461228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:07.9462133Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:07.9522862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:07.9525800Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:08.3270385Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:08.3892620Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:10.1594786Z ok (3.708s) 2022-03-17T19:57:10.1595215Z 2022-03-17T19:57:10.1596454Z ---------------------------------------------------------------------- 2022-03-17T19:57:10.1597456Z Ran 1 test in 3.708s 2022-03-17T19:57:10.1597934Z 2022-03-17T19:57:10.1598203Z OK 2022-03-17T19:57:10.1598680Z 2022-03-17T19:57:10.1602855Z Generating XML reports... 2022-03-17T19:57:10.1631005Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195706.xml 2022-03-17T19:57:12.7635263Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:57:12.7644709Z 2022-03-17T19:57:12.7645336Z Running tests... 2022-03-17T19:57:12.7646513Z ---------------------------------------------------------------------- 2022-03-17T19:57:13.3569654Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57760 2022-03-17T19:57:13.3606740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57761 2022-03-17T19:57:14.2745140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:14.2786976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:14.6763454Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:14.7297868Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:26.3925377Z ok (13.627s) 2022-03-17T19:57:26.3925814Z 2022-03-17T19:57:26.3926740Z ---------------------------------------------------------------------- 2022-03-17T19:57:26.3927511Z Ran 1 test in 13.628s 2022-03-17T19:57:26.3927870Z 2022-03-17T19:57:26.3928058Z OK 2022-03-17T19:57:26.3928347Z 2022-03-17T19:57:26.3928619Z Generating XML reports... 2022-03-17T19:57:26.4011137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195712.xml 2022-03-17T19:57:28.8695104Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:57:28.8708001Z 2022-03-17T19:57:28.8708273Z Running tests... 2022-03-17T19:57:28.8708708Z ---------------------------------------------------------------------- 2022-03-17T19:57:29.4647189Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57979 2022-03-17T19:57:29.4686559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57980 2022-03-17T19:57:30.3318300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:30.4020333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:32.5778721Z skip (3.706s) 2022-03-17T19:57:32.5779420Z 2022-03-17T19:57:32.5780412Z ---------------------------------------------------------------------- 2022-03-17T19:57:32.5781218Z Ran 1 test in 3.707s 2022-03-17T19:57:32.5781620Z 2022-03-17T19:57:32.5781866Z OK (skipped=1) 2022-03-17T19:57:32.5782220Z 2022-03-17T19:57:32.5782501Z Generating XML reports... 2022-03-17T19:57:32.5861089Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195728.xml 2022-03-17T19:57:35.0488460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:57:35.0502898Z 2022-03-17T19:57:35.0503766Z Running tests... 2022-03-17T19:57:35.0504957Z ---------------------------------------------------------------------- 2022-03-17T19:57:35.6449156Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58183 2022-03-17T19:57:35.6487121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58184 2022-03-17T19:57:36.5172643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:36.5387446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:38.4559064Z skip (3.405s) 2022-03-17T19:57:38.4559497Z 2022-03-17T19:57:38.4560469Z ---------------------------------------------------------------------- 2022-03-17T19:57:38.4561406Z Ran 1 test in 3.406s 2022-03-17T19:57:38.4561820Z 2022-03-17T19:57:38.4562233Z OK (skipped=1) 2022-03-17T19:57:38.4562604Z 2022-03-17T19:57:38.4570015Z Generating XML reports... 2022-03-17T19:57:38.4642648Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195735.xml 2022-03-17T19:57:40.9879408Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:57:40.9894920Z 2022-03-17T19:57:40.9895167Z Running tests... 2022-03-17T19:57:40.9895634Z ---------------------------------------------------------------------- 2022-03-17T19:57:41.5895942Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58387 2022-03-17T19:57:41.5941305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58388 2022-03-17T19:57:42.5492112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:42.5494995Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5498033Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5500724Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5503341Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5505940Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5508209Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5782296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:42.5784433Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5787070Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5789736Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5792317Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5794832Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:42.5797438Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:57:44.5020165Z ok (3.512s) 2022-03-17T19:57:44.5020529Z 2022-03-17T19:57:44.5021648Z ---------------------------------------------------------------------- 2022-03-17T19:57:44.5022676Z Ran 1 test in 3.512s 2022-03-17T19:57:44.5023141Z 2022-03-17T19:57:44.5023362Z OK 2022-03-17T19:57:44.5023745Z 2022-03-17T19:57:44.5024028Z Generating XML reports... 2022-03-17T19:57:44.5106165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195740.xml 2022-03-17T19:57:46.9359893Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:57:46.9371604Z 2022-03-17T19:57:46.9371959Z Running tests... 2022-03-17T19:57:46.9372407Z ---------------------------------------------------------------------- 2022-03-17T19:57:47.5292187Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58591 2022-03-17T19:57:47.5334597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58592 2022-03-17T19:57:48.3887440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:48.4401613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:48.8230971Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:57:48.8705954Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:00.7599032Z ok (13.822s) 2022-03-17T19:58:00.7599646Z 2022-03-17T19:58:00.7600973Z ---------------------------------------------------------------------- 2022-03-17T19:58:00.7602299Z Ran 1 test in 13.822s 2022-03-17T19:58:00.7602867Z 2022-03-17T19:58:00.7603122Z OK 2022-03-17T19:58:00.7603605Z 2022-03-17T19:58:00.7604068Z Generating XML reports... 2022-03-17T19:58:00.7681038Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195746.xml 2022-03-17T19:58:03.2795465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:58:03.2806921Z 2022-03-17T19:58:03.2807376Z Running tests... 2022-03-17T19:58:03.2808348Z ---------------------------------------------------------------------- 2022-03-17T19:58:03.8721776Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58810 2022-03-17T19:58:03.8756917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58811 2022-03-17T19:58:04.7793893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:04.7918298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:05.1607330Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:05.2247441Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:17.1015771Z ok (13.820s) 2022-03-17T19:58:17.1016358Z 2022-03-17T19:58:17.1017532Z ---------------------------------------------------------------------- 2022-03-17T19:58:17.1018524Z Ran 1 test in 13.820s 2022-03-17T19:58:17.1018901Z 2022-03-17T19:58:17.1019095Z OK 2022-03-17T19:58:17.1019395Z 2022-03-17T19:58:17.1019667Z Generating XML reports... 2022-03-17T19:58:17.1096335Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195803.xml 2022-03-17T19:58:19.5938690Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:58:19.5950266Z 2022-03-17T19:58:19.5950775Z Running tests... 2022-03-17T19:58:19.5951978Z ---------------------------------------------------------------------- 2022-03-17T19:58:20.2012150Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59029 2022-03-17T19:58:20.2064844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59030 2022-03-17T19:58:21.1315085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:21.1447193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:21.5120104Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:21.5658207Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:31.0410869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:58:31.0412096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:58:33.6328249Z ok (14.037s) 2022-03-17T19:58:33.6328738Z 2022-03-17T19:58:33.6330460Z ---------------------------------------------------------------------- 2022-03-17T19:58:33.6331350Z Ran 1 test in 14.038s 2022-03-17T19:58:33.6331702Z 2022-03-17T19:58:33.6331897Z OK 2022-03-17T19:58:33.6332219Z 2022-03-17T19:58:33.6332512Z Generating XML reports... 2022-03-17T19:58:33.6412195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195819.xml 2022-03-17T19:58:36.1182908Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:58:36.1194740Z 2022-03-17T19:58:36.1195418Z Running tests... 2022-03-17T19:58:36.1196592Z ---------------------------------------------------------------------- 2022-03-17T19:58:36.7128886Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59248 2022-03-17T19:58:36.7167471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59249 2022-03-17T19:58:37.5942067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:37.5946691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:37.9763483Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:38.0266797Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:58:47.3330828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:58:47.4160008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:58:49.9477981Z ok (13.828s) 2022-03-17T19:58:49.9478429Z 2022-03-17T19:58:49.9479561Z ---------------------------------------------------------------------- 2022-03-17T19:58:49.9480660Z Ran 1 test in 13.828s 2022-03-17T19:58:49.9481114Z 2022-03-17T19:58:49.9481304Z OK 2022-03-17T19:58:49.9484164Z 2022-03-17T19:58:49.9484981Z Generating XML reports... 2022-03-17T19:58:49.9550298Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195836.xml 2022-03-17T19:58:52.4075831Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:58:52.4089176Z 2022-03-17T19:58:52.4089816Z Running tests... 2022-03-17T19:58:52.4091241Z ---------------------------------------------------------------------- 2022-03-17T19:58:53.0009725Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59467 2022-03-17T19:58:53.0048101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59468 2022-03-17T19:58:53.9225056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:53.9747637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:55.8121061Z skip (3.403s) 2022-03-17T19:58:55.8121464Z 2022-03-17T19:58:55.8122855Z ---------------------------------------------------------------------- 2022-03-17T19:58:55.8123851Z Ran 1 test in 3.403s 2022-03-17T19:58:55.8124326Z 2022-03-17T19:58:55.8124777Z OK (skipped=1) 2022-03-17T19:58:55.8125174Z 2022-03-17T19:58:55.8125424Z Generating XML reports... 2022-03-17T19:58:55.8206475Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195852.xml 2022-03-17T19:58:58.2226728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:58:58.2238142Z 2022-03-17T19:58:58.2238402Z Running tests... 2022-03-17T19:58:58.2238829Z ---------------------------------------------------------------------- 2022-03-17T19:58:58.8165553Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59671 2022-03-17T19:58:58.8196855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59672 2022-03-17T19:58:59.6784764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:59.7195649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:01.5282647Z skip (3.304s) 2022-03-17T19:59:01.5283063Z 2022-03-17T19:59:01.5283934Z ---------------------------------------------------------------------- 2022-03-17T19:59:01.5284904Z Ran 1 test in 3.304s 2022-03-17T19:59:01.5285368Z 2022-03-17T19:59:01.5285685Z OK (skipped=1) 2022-03-17T19:59:01.5286172Z 2022-03-17T19:59:01.5286503Z Generating XML reports... 2022-03-17T19:59:01.5368387Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195858.xml 2022-03-17T19:59:03.9470791Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:03.9486722Z 2022-03-17T19:59:03.9487392Z Running tests... 2022-03-17T19:59:03.9489289Z ---------------------------------------------------------------------- 2022-03-17T19:59:04.5354461Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59875 2022-03-17T19:59:04.5387701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59876 2022-03-17T19:59:05.4479638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:05.4537250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:07.3457938Z ok (3.397s) 2022-03-17T19:59:07.3458523Z 2022-03-17T19:59:07.3459817Z ---------------------------------------------------------------------- 2022-03-17T19:59:07.3460834Z Ran 1 test in 3.397s 2022-03-17T19:59:07.3461300Z 2022-03-17T19:59:07.3461497Z OK 2022-03-17T19:59:07.3461804Z 2022-03-17T19:59:07.3462078Z Generating XML reports... 2022-03-17T19:59:07.3538971Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195903.xml 2022-03-17T19:59:09.8050466Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:09.8061435Z 2022-03-17T19:59:09.8062092Z Running tests... 2022-03-17T19:59:09.8063357Z ---------------------------------------------------------------------- 2022-03-17T19:59:10.3985319Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60083 2022-03-17T19:59:10.4017537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60084 2022-03-17T19:59:11.2677966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:11.2752426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:13.6087660Z skip (3.802s) 2022-03-17T19:59:13.6088031Z 2022-03-17T19:59:13.6089017Z ---------------------------------------------------------------------- 2022-03-17T19:59:13.6089976Z Ran 1 test in 3.803s 2022-03-17T19:59:13.6090450Z 2022-03-17T19:59:13.6090834Z OK (skipped=1) 2022-03-17T19:59:13.6091183Z 2022-03-17T19:59:13.6093318Z Generating XML reports... 2022-03-17T19:59:13.6172829Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195909.xml 2022-03-17T19:59:16.1090559Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:16.1099967Z 2022-03-17T19:59:16.1100601Z Running tests... 2022-03-17T19:59:16.1101781Z ---------------------------------------------------------------------- 2022-03-17T19:59:16.7010492Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60287 2022-03-17T19:59:16.7050665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60288 2022-03-17T19:59:17.5862795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:17.6234826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:18.0077969Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:59:18.0720018Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:59:27.5315776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:59:27.5317021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:59:30.0288384Z ok (13.918s) 2022-03-17T19:59:30.0288721Z 2022-03-17T19:59:30.0295314Z ---------------------------------------------------------------------- 2022-03-17T19:59:30.0296144Z Ran 1 test in 13.918s 2022-03-17T19:59:30.0296531Z 2022-03-17T19:59:30.0297233Z OK 2022-03-17T19:59:30.0297542Z 2022-03-17T19:59:30.0297826Z Generating XML reports... 2022-03-17T19:59:30.0370174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195916.xml 2022-03-17T19:59:33.0077449Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:33.0091866Z 2022-03-17T19:59:33.0092408Z Running tests... 2022-03-17T19:59:33.0093624Z ---------------------------------------------------------------------- 2022-03-17T19:59:33.6063170Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60506 2022-03-17T19:59:33.6111430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60507 2022-03-17T19:59:34.4966609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:34.5472481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:34.9280572Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:59:34.9836713Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:59:44.3463099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:59:44.3464750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:59:46.8419217Z ok (13.832s) 2022-03-17T19:59:46.8419817Z 2022-03-17T19:59:46.8420840Z ---------------------------------------------------------------------- 2022-03-17T19:59:46.8421656Z Ran 1 test in 13.833s 2022-03-17T19:59:46.8422028Z 2022-03-17T19:59:46.8422218Z OK 2022-03-17T19:59:46.8422503Z 2022-03-17T19:59:46.8423319Z Generating XML reports... 2022-03-17T19:59:46.8493216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195933.xml 2022-03-17T19:59:49.4016152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:49.4026428Z 2022-03-17T19:59:49.4027057Z Running tests... 2022-03-17T19:59:49.4028247Z ---------------------------------------------------------------------- 2022-03-17T19:59:50.0101354Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60725 2022-03-17T19:59:50.0150596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60726 2022-03-17T19:59:50.9250857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:50.9436645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:51.0085166Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T19:59:51.0676855Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:09.1524831Z ok (19.749s) 2022-03-17T20:00:09.1525245Z 2022-03-17T20:00:09.1526200Z ---------------------------------------------------------------------- 2022-03-17T20:00:09.1526963Z Ran 1 test in 19.750s 2022-03-17T20:00:09.1533264Z 2022-03-17T20:00:09.1533655Z OK 2022-03-17T20:00:09.1534130Z 2022-03-17T20:00:09.1534483Z Generating XML reports... 2022-03-17T20:00:09.1607307Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195949.xml 2022-03-17T20:00:11.6309239Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:11.6322405Z 2022-03-17T20:00:11.6322886Z Running tests... 2022-03-17T20:00:11.6323866Z ---------------------------------------------------------------------- 2022-03-17T20:00:11.6346998Z test_no_grad (__main__.DistributedDataParallelTest) 2022-03-17T20:00:12.2256704Z Note: this test can be sped up by only running it on a CPU module ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60944 2022-03-17T20:00:12.2304728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60945 2022-03-17T20:00:13.1225098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:13.1549632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:13.5018219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:13.5548729Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:25.8571446Z ok (14.224s) 2022-03-17T20:00:25.8571906Z 2022-03-17T20:00:25.8573164Z ---------------------------------------------------------------------- 2022-03-17T20:00:25.8573945Z Ran 1 test in 14.225s 2022-03-17T20:00:25.8574311Z 2022-03-17T20:00:25.8574501Z OK 2022-03-17T20:00:25.8574796Z 2022-03-17T20:00:25.8582688Z Generating XML reports... 2022-03-17T20:00:25.8651520Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200011.xml 2022-03-17T20:00:28.2745339Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:28.2758039Z 2022-03-17T20:00:28.2758592Z Running tests... 2022-03-17T20:00:28.2759585Z ---------------------------------------------------------------------- 2022-03-17T20:00:28.8686324Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61159 2022-03-17T20:00:28.8725654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61160 2022-03-17T20:00:29.7607985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:29.8229131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:30.1441047Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:30.2075771Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:31.8817740Z ok (3.605s) 2022-03-17T20:00:31.8818163Z 2022-03-17T20:00:31.8819163Z ---------------------------------------------------------------------- 2022-03-17T20:00:31.8819918Z Ran 1 test in 3.606s 2022-03-17T20:00:31.8820299Z 2022-03-17T20:00:31.8820485Z OK 2022-03-17T20:00:31.8820779Z 2022-03-17T20:00:31.8821867Z Generating XML reports... 2022-03-17T20:00:31.8901144Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200028.xml 2022-03-17T20:00:34.3548536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:34.3559989Z 2022-03-17T20:00:34.3560699Z Running tests... 2022-03-17T20:00:34.3562443Z ---------------------------------------------------------------------- 2022-03-17T20:00:34.9502579Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61374 2022-03-17T20:00:34.9542173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61375 2022-03-17T20:00:35.8734161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:35.8734628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:00:35.9081105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:35.9082824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:00:35.9083646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:00:35.9142155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:00:37.6611423Z ok (3.305s) 2022-03-17T20:00:37.6611969Z 2022-03-17T20:00:37.6613155Z ---------------------------------------------------------------------- 2022-03-17T20:00:37.6614138Z Ran 1 test in 3.305s 2022-03-17T20:00:37.6614603Z 2022-03-17T20:00:37.6614839Z OK 2022-03-17T20:00:37.6615208Z 2022-03-17T20:00:37.6615518Z Generating XML reports... 2022-03-17T20:00:37.6697601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200034.xml 2022-03-17T20:00:40.2965168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:40.2977007Z 2022-03-17T20:00:40.2977304Z Running tests... 2022-03-17T20:00:40.8909402Z ---------------------------------------------------------------------- 2022-03-17T20:00:40.8910875Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61582 2022-03-17T20:00:40.8949620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61583 2022-03-17T20:00:41.7752636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:41.7753666Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:41.8258846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:41.8259765Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:42.1537538Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:42.2149479Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:42.4044051Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:42.4047114Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:42.4108778Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:42.4112309Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:42.4171830Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:42.4174144Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:44.1021131Z ok (3.804s) 2022-03-17T20:00:44.1021774Z 2022-03-17T20:00:44.1022945Z ---------------------------------------------------------------------- 2022-03-17T20:00:44.1023962Z Ran 1 test in 3.804s 2022-03-17T20:00:44.1024455Z 2022-03-17T20:00:44.1024643Z OK 2022-03-17T20:00:44.1024968Z 2022-03-17T20:00:44.1025286Z Generating XML reports... 2022-03-17T20:00:44.1102438Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200040.xml 2022-03-17T20:00:46.6335572Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:46.6348492Z 2022-03-17T20:00:46.6349201Z Running tests... 2022-03-17T20:00:46.6350658Z ---------------------------------------------------------------------- 2022-03-17T20:00:47.2270062Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61801 2022-03-17T20:00:47.2306998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61802 2022-03-17T20:00:48.1609533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:48.1611293Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.2296778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:48.2299040Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.5446217Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:48.6149698Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:00:48.8027019Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.8029878Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.8093669Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.8096308Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.8149598Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:48.8152229Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:50.5377238Z ok (3.903s) 2022-03-17T20:00:50.5377895Z 2022-03-17T20:00:50.5379067Z ---------------------------------------------------------------------- 2022-03-17T20:00:50.5380056Z Ran 1 test in 3.903s 2022-03-17T20:00:50.5380547Z 2022-03-17T20:00:50.5380770Z OK 2022-03-17T20:00:50.5381129Z 2022-03-17T20:00:50.5381600Z Generating XML reports... 2022-03-17T20:00:50.5460350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200046.xml 2022-03-17T20:00:53.3335947Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:53.3348379Z 2022-03-17T20:00:53.3348674Z Running tests... 2022-03-17T20:00:53.3349134Z ---------------------------------------------------------------------- 2022-03-17T20:00:53.9301331Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62020 2022-03-17T20:00:53.9334547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62021 2022-03-17T20:00:53.9386258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62022 2022-03-17T20:00:54.8405501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:00:54.8512558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:54.8557573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:56.6472782Z skip (3.312s) 2022-03-17T20:00:56.6473300Z 2022-03-17T20:00:56.6474645Z ---------------------------------------------------------------------- 2022-03-17T20:00:56.6475664Z Ran 1 test in 3.312s 2022-03-17T20:00:56.6476092Z 2022-03-17T20:00:56.6476415Z OK (skipped=1) 2022-03-17T20:00:56.6476833Z 2022-03-17T20:00:56.6477166Z Generating XML reports... 2022-03-17T20:00:56.6557288Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200053.xml 2022-03-17T20:00:59.5404445Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:59.5419001Z 2022-03-17T20:00:59.5419689Z Running tests... 2022-03-17T20:00:59.5420123Z ---------------------------------------------------------------------- 2022-03-17T20:01:00.1352788Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62292 2022-03-17T20:01:00.1386362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62293 2022-03-17T20:01:00.1437580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62294 2022-03-17T20:01:01.0270663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:01.0505702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:01.1017248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:01:02.9511679Z skip (3.409s) 2022-03-17T20:01:02.9512117Z 2022-03-17T20:01:02.9513500Z ---------------------------------------------------------------------- 2022-03-17T20:01:02.9514690Z Ran 1 test in 3.409s 2022-03-17T20:01:02.9515058Z 2022-03-17T20:01:02.9515308Z OK (skipped=1) 2022-03-17T20:01:02.9515668Z 2022-03-17T20:01:02.9515937Z Generating XML reports... 2022-03-17T20:01:02.9595839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200059.xml 2022-03-17T20:01:05.6761784Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:05.6775439Z 2022-03-17T20:01:05.6775724Z Running tests... 2022-03-17T20:01:05.6776160Z ---------------------------------------------------------------------- 2022-03-17T20:01:05.6779802Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip (0.001s) 2022-03-17T20:01:05.6780605Z 2022-03-17T20:01:05.6781579Z ---------------------------------------------------------------------- 2022-03-17T20:01:05.6782473Z Ran 1 test in 0.001s 2022-03-17T20:01:05.6782840Z 2022-03-17T20:01:05.6783087Z OK (skipped=1) 2022-03-17T20:01:05.6783438Z 2022-03-17T20:01:05.6783695Z Generating XML reports... 2022-03-17T20:01:05.6817513Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200105.xml 2022-03-17T20:01:08.0497954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:08.0510176Z 2022-03-17T20:01:08.0510680Z Running tests... 2022-03-17T20:01:08.0511129Z ---------------------------------------------------------------------- 2022-03-17T20:01:08.6430055Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62631 2022-03-17T20:01:08.6465693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62632 2022-03-17T20:01:08.6513437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62633 2022-03-17T20:01:09.5451342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:01:09.5607444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:09.5856722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:11.6594443Z skip (3.608s) 2022-03-17T20:01:11.6595136Z 2022-03-17T20:01:11.6596405Z ---------------------------------------------------------------------- 2022-03-17T20:01:11.6597238Z Ran 1 test in 3.608s 2022-03-17T20:01:11.6597713Z 2022-03-17T20:01:11.6598041Z OK (skipped=1) 2022-03-17T20:01:11.6598593Z 2022-03-17T20:01:11.6598865Z Generating XML reports... 2022-03-17T20:01:11.6682169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200108.xml 2022-03-17T20:01:14.1283539Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:14.1295449Z 2022-03-17T20:01:14.1296329Z Running tests... 2022-03-17T20:01:14.1296783Z ---------------------------------------------------------------------- 2022-03-17T20:01:14.7240463Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62903 2022-03-17T20:01:14.7319806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62904 2022-03-17T20:01:14.7401613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62905 2022-03-17T20:01:15.6209178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:01:15.6714438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:15.6714884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:17.4478344Z ok (3.318s) 2022-03-17T20:01:17.4478744Z 2022-03-17T20:01:17.4479891Z ---------------------------------------------------------------------- 2022-03-17T20:01:17.4480845Z Ran 1 test in 3.318s 2022-03-17T20:01:17.4481290Z 2022-03-17T20:01:17.4481524Z OK 2022-03-17T20:01:17.4482121Z 2022-03-17T20:01:17.4482480Z Generating XML reports... 2022-03-17T20:01:17.4552764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200114.xml 2022-03-17T20:01:19.9753943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:19.9770850Z 2022-03-17T20:01:19.9771068Z Running tests... 2022-03-17T20:01:19.9771508Z ---------------------------------------------------------------------- 2022-03-17T20:01:20.5689731Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63175 2022-03-17T20:01:20.5770979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63176 2022-03-17T20:01:20.5832204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63177 2022-03-17T20:01:21.4907825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:21.5150164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:21.5172650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:01:23.3920145Z ok (3.415s) 2022-03-17T20:01:23.3920515Z 2022-03-17T20:01:23.3921570Z ---------------------------------------------------------------------- 2022-03-17T20:01:23.3922546Z Ran 1 test in 3.415s 2022-03-17T20:01:23.3922920Z 2022-03-17T20:01:23.3924006Z OK 2022-03-17T20:01:23.3924435Z 2022-03-17T20:01:23.3924728Z Generating XML reports... 2022-03-17T20:01:23.4005296Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200119.xml 2022-03-17T20:01:25.8482979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:25.8495896Z 2022-03-17T20:01:25.8496252Z Running tests... 2022-03-17T20:01:25.8496785Z ---------------------------------------------------------------------- 2022-03-17T20:01:26.4415974Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63447 2022-03-17T20:01:26.4456225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63448 2022-03-17T20:01:26.4512633Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63449 2022-03-17T20:01:27.3392539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:27.3949279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:01:27.4380717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:29.2605106Z ok (3.410s) 2022-03-17T20:01:29.2605530Z 2022-03-17T20:01:29.2606937Z ---------------------------------------------------------------------- 2022-03-17T20:01:29.2607729Z Ran 1 test in 3.411s 2022-03-17T20:01:29.2608096Z 2022-03-17T20:01:29.2608285Z OK 2022-03-17T20:01:29.2608570Z 2022-03-17T20:01:29.2608843Z Generating XML reports... 2022-03-17T20:01:29.2686591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200125.xml 2022-03-17T20:01:31.7179698Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:31.7189678Z 2022-03-17T20:01:31.7190270Z Running tests... 2022-03-17T20:01:31.7191445Z ---------------------------------------------------------------------- 2022-03-17T20:01:31.7205887Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip (0.002s) 2022-03-17T20:01:31.7206693Z 2022-03-17T20:01:31.7207565Z ---------------------------------------------------------------------- 2022-03-17T20:01:31.7208534Z Ran 1 test in 0.002s 2022-03-17T20:01:31.7208978Z 2022-03-17T20:01:31.7209268Z OK (skipped=1) 2022-03-17T20:01:31.7209628Z 2022-03-17T20:01:31.7209896Z Generating XML reports... 2022-03-17T20:01:31.7237533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200131.xml 2022-03-17T20:01:34.3085929Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:34.3099572Z 2022-03-17T20:01:34.3099890Z Running tests... 2022-03-17T20:01:34.3101012Z ---------------------------------------------------------------------- 2022-03-17T20:01:34.9039081Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63786 2022-03-17T20:01:34.9080947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63787 2022-03-17T20:01:34.9132192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63788 2022-03-17T20:01:35.7857877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:35.8620444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:01:35.8700678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:37.7224830Z skip (3.412s) 2022-03-17T20:01:37.7225450Z 2022-03-17T20:01:37.7226618Z ---------------------------------------------------------------------- 2022-03-17T20:01:37.7227627Z Ran 1 test in 3.413s 2022-03-17T20:01:37.7228092Z 2022-03-17T20:01:37.7228403Z OK (skipped=1) 2022-03-17T20:01:37.7228805Z 2022-03-17T20:01:37.7229114Z Generating XML reports... 2022-03-17T20:01:37.7303817Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200134.xml 2022-03-17T20:01:40.3710819Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:40.3723102Z 2022-03-17T20:01:40.3723422Z Running tests... 2022-03-17T20:01:40.3724523Z ---------------------------------------------------------------------- 2022-03-17T20:01:40.3729387Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip (0.001s) 2022-03-17T20:01:40.3730083Z 2022-03-17T20:01:40.3730844Z ---------------------------------------------------------------------- 2022-03-17T20:01:40.3731671Z Ran 1 test in 0.001s 2022-03-17T20:01:40.3732058Z 2022-03-17T20:01:40.3732302Z OK (skipped=1) 2022-03-17T20:01:40.3732696Z 2022-03-17T20:01:40.3732989Z Generating XML reports... 2022-03-17T20:01:40.3762081Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220317200140.xml 2022-03-17T20:01:42.9826998Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:42.9839912Z 2022-03-17T20:01:42.9840397Z Running tests... 2022-03-17T20:01:42.9840947Z ---------------------------------------------------------------------- 2022-03-17T20:01:43.5768104Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64125 2022-03-17T20:01:43.5811162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64126 2022-03-17T20:01:44.4464194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:44.4464783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:01:44.5058099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:44.5058933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:01:44.5059782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:01:44.5079836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:01:46.7907788Z ok (3.806s) 2022-03-17T20:01:46.7908168Z 2022-03-17T20:01:46.7909361Z ---------------------------------------------------------------------- 2022-03-17T20:01:46.7910201Z Ran 1 test in 3.807s 2022-03-17T20:01:46.7910567Z 2022-03-17T20:01:46.7910754Z OK 2022-03-17T20:01:46.7911030Z 2022-03-17T20:01:46.7911302Z Generating XML reports... 2022-03-17T20:01:46.7985686Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200142.xml 2022-03-17T20:01:49.6375460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:49.6387666Z 2022-03-17T20:01:49.6388716Z Running tests... 2022-03-17T20:01:49.6389890Z ---------------------------------------------------------------------- 2022-03-17T20:01:50.2326179Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64333 2022-03-17T20:01:50.2363510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64334 2022-03-17T20:01:51.1195694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:51.1196172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:01:51.1344777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:51.1346059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:01:51.1347706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:01:51.1403988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:01:51.5161249Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:01:51.5672178Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:01:53.4455985Z ok (3.806s) 2022-03-17T20:01:53.4456373Z 2022-03-17T20:01:53.4457231Z ---------------------------------------------------------------------- 2022-03-17T20:01:53.4458019Z Ran 1 test in 3.806s 2022-03-17T20:01:53.4458384Z 2022-03-17T20:01:53.4458566Z OK 2022-03-17T20:01:53.4458868Z 2022-03-17T20:01:53.4459138Z Generating XML reports... 2022-03-17T20:01:53.4536659Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200149.xml 2022-03-17T20:01:55.9338867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:55.9350635Z 2022-03-17T20:01:55.9350916Z Running tests... 2022-03-17T20:01:55.9351396Z ---------------------------------------------------------------------- 2022-03-17T20:01:56.5288241Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64548 2022-03-17T20:01:56.5327081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64549 2022-03-17T20:01:57.4386634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:57.4387101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:01:57.4534474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:57.4535566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:01:57.4537157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:01:57.4596481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:01:57.8448009Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:01:57.9111649Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:00.2409378Z ok (4.305s) 2022-03-17T20:02:00.2409887Z 2022-03-17T20:02:00.2411069Z ---------------------------------------------------------------------- 2022-03-17T20:02:00.2412027Z Ran 1 test in 4.306s 2022-03-17T20:02:00.2412471Z 2022-03-17T20:02:00.2412730Z OK 2022-03-17T20:02:00.2420078Z 2022-03-17T20:02:00.2421195Z Generating XML reports... 2022-03-17T20:02:00.2491573Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200155.xml 2022-03-17T20:02:02.6635437Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:02.6649753Z 2022-03-17T20:02:02.6650047Z Running tests... 2022-03-17T20:02:02.6650483Z ---------------------------------------------------------------------- 2022-03-17T20:02:03.2600256Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64763 2022-03-17T20:02:03.2637222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64764 2022-03-17T20:02:04.1905780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:04.1906817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:04.1948758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:04.1949791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:04.1951342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:04.2015511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:04.5763239Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:04.6275800Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:06.3756152Z ok (3.710s) 2022-03-17T20:02:06.3760488Z 2022-03-17T20:02:06.3762343Z ---------------------------------------------------------------------- 2022-03-17T20:02:06.3763235Z Ran 1 test in 3.710s 2022-03-17T20:02:06.3763612Z 2022-03-17T20:02:06.3763797Z OK 2022-03-17T20:02:06.3764140Z 2022-03-17T20:02:06.3764410Z Generating XML reports... 2022-03-17T20:02:06.3841220Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200202.xml 2022-03-17T20:02:08.7887800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:08.7898206Z 2022-03-17T20:02:08.7898727Z Running tests... 2022-03-17T20:02:08.7899886Z ---------------------------------------------------------------------- 2022-03-17T20:02:09.3842776Z test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64978 2022-03-17T20:02:09.3877890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64979 2022-03-17T20:02:10.2659157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:10.2660366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:10.3071606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:10.3072855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:10.3074702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:10.3169862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:10.6879779Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:10.7420006Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:12.3954898Z ok (3.605s) 2022-03-17T20:02:12.3955296Z 2022-03-17T20:02:12.3956216Z ---------------------------------------------------------------------- 2022-03-17T20:02:12.3957145Z Ran 1 test in 3.605s 2022-03-17T20:02:12.3957641Z 2022-03-17T20:02:12.3958267Z OK 2022-03-17T20:02:12.3958577Z 2022-03-17T20:02:12.3958837Z Generating XML reports... 2022-03-17T20:02:12.4037164Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200208.xml 2022-03-17T20:02:14.9229987Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:14.9242034Z 2022-03-17T20:02:14.9242421Z Running tests... 2022-03-17T20:02:14.9243016Z ---------------------------------------------------------------------- 2022-03-17T20:02:15.5169878Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65193 2022-03-17T20:02:15.5208865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65194 2022-03-17T20:02:16.3800900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:16.3802167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:16.4343585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:16.4344840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:16.4346798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:16.4416931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:16.8227212Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:16.8831222Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:18.6298125Z ok (3.705s) 2022-03-17T20:02:18.6298657Z 2022-03-17T20:02:18.6299644Z ---------------------------------------------------------------------- 2022-03-17T20:02:18.6300575Z Ran 1 test in 3.706s 2022-03-17T20:02:18.6301064Z 2022-03-17T20:02:18.6301286Z OK 2022-03-17T20:02:18.6301592Z 2022-03-17T20:02:18.6301873Z Generating XML reports... 2022-03-17T20:02:18.6379651Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200214.xml 2022-03-17T20:02:21.0298822Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:21.0308922Z 2022-03-17T20:02:21.0309232Z Running tests... 2022-03-17T20:02:21.0310425Z ---------------------------------------------------------------------- 2022-03-17T20:02:21.6253504Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65408 2022-03-17T20:02:21.6295343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65409 2022-03-17T20:02:22.5144322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:22.5144803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:22.5635309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:22.5635884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:22.5636894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:22.5658556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:22.6081466Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:22.6670458Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:25.4391621Z ok (4.408s) 2022-03-17T20:02:25.4391995Z 2022-03-17T20:02:25.4392883Z ---------------------------------------------------------------------- 2022-03-17T20:02:25.4394185Z Ran 1 test in 4.408s 2022-03-17T20:02:25.4394547Z 2022-03-17T20:02:25.4394717Z OK 2022-03-17T20:02:25.4395010Z 2022-03-17T20:02:25.4395281Z Generating XML reports... 2022-03-17T20:02:25.4471532Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200221.xml 2022-03-17T20:02:27.8330536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:27.8342400Z 2022-03-17T20:02:27.8342937Z Running tests... 2022-03-17T20:02:27.8343996Z ---------------------------------------------------------------------- 2022-03-17T20:02:28.4307821Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65623 2022-03-17T20:02:28.4341242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65624 2022-03-17T20:02:29.3135492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:29.3136944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:29.3507749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:29.3509237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:29.3511179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:29.3547370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:31.3432263Z ok (3.508s) 2022-03-17T20:02:31.3432641Z 2022-03-17T20:02:31.3433800Z ---------------------------------------------------------------------- 2022-03-17T20:02:31.3434710Z Ran 1 test in 3.509s 2022-03-17T20:02:31.3435080Z 2022-03-17T20:02:31.3435266Z OK 2022-03-17T20:02:31.3435567Z 2022-03-17T20:02:31.3436692Z Generating XML reports... 2022-03-17T20:02:31.3516498Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200227.xml 2022-03-17T20:02:33.8157314Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:33.8171218Z 2022-03-17T20:02:33.8171598Z Running tests... 2022-03-17T20:02:33.8172121Z ---------------------------------------------------------------------- 2022-03-17T20:02:34.4108323Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65831 2022-03-17T20:02:34.4150483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65832 2022-03-17T20:02:35.3120314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:35.3121127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:35.3461396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:35.3461954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:35.3462938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:35.3533783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:35.7347187Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:35.7883772Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:37.5224657Z ok (3.705s) 2022-03-17T20:02:37.5225010Z 2022-03-17T20:02:37.5225992Z ---------------------------------------------------------------------- 2022-03-17T20:02:37.5226757Z Ran 1 test in 3.705s 2022-03-17T20:02:37.5227120Z 2022-03-17T20:02:37.5227745Z OK 2022-03-17T20:02:37.5228050Z 2022-03-17T20:02:37.5232043Z Generating XML reports... 2022-03-17T20:02:37.5303095Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200233.xml 2022-03-17T20:02:40.0273564Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:40.0284361Z 2022-03-17T20:02:40.0285146Z Running tests... 2022-03-17T20:02:40.0286396Z ---------------------------------------------------------------------- 2022-03-17T20:02:40.6197471Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66046 2022-03-17T20:02:40.6235182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66047 2022-03-17T20:02:41.4747312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:41.4750589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:41.5141376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:41.5144169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:41.5145476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:41.5161081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:41.9669694Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:42.0244923Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:44.2328584Z ok (4.204s) 2022-03-17T20:02:44.2328997Z 2022-03-17T20:02:44.2330333Z ---------------------------------------------------------------------- 2022-03-17T20:02:44.2331261Z Ran 1 test in 4.204s 2022-03-17T20:02:44.2331675Z 2022-03-17T20:02:44.2331867Z OK 2022-03-17T20:02:44.2332155Z 2022-03-17T20:02:44.2332451Z Generating XML reports... 2022-03-17T20:02:44.2413636Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200240.xml 2022-03-17T20:02:46.6405145Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:46.6418950Z 2022-03-17T20:02:46.6419240Z Running tests... 2022-03-17T20:02:46.6420820Z ---------------------------------------------------------------------- 2022-03-17T20:02:47.2368953Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66261 2022-03-17T20:02:47.2445126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66262 2022-03-17T20:02:48.1398996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:48.1400878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:48.1696798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:48.1699264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:48.1700343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:48.1709035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:48.5458187Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:48.6024172Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:02:50.3525392Z ok (3.710s) 2022-03-17T20:02:50.3525760Z 2022-03-17T20:02:50.3526628Z ---------------------------------------------------------------------- 2022-03-17T20:02:50.3527819Z Ran 1 test in 3.711s 2022-03-17T20:02:50.3528208Z 2022-03-17T20:02:50.3528395Z OK 2022-03-17T20:02:50.3528670Z 2022-03-17T20:02:50.3528951Z Generating XML reports... 2022-03-17T20:02:50.3609506Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200246.xml 2022-03-17T20:02:52.9857441Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:52.9870228Z 2022-03-17T20:02:52.9870485Z Running tests... 2022-03-17T20:02:52.9871410Z ---------------------------------------------------------------------- 2022-03-17T20:02:53.5808700Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66476 2022-03-17T20:02:53.5846361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66477 2022-03-17T20:02:54.4458621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:54.4460125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:02:54.5106266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:54.5107704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:02:54.5109462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:54.5171029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:02:57.1949787Z ok (4.207s) 2022-03-17T20:02:57.1950188Z 2022-03-17T20:02:57.1951061Z ---------------------------------------------------------------------- 2022-03-17T20:02:57.1951843Z Ran 1 test in 4.208s 2022-03-17T20:02:57.1952226Z 2022-03-17T20:02:57.1952431Z OK 2022-03-17T20:02:57.1952734Z 2022-03-17T20:02:57.1953033Z Generating XML reports... 2022-03-17T20:02:57.2033807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200252.xml 2022-03-17T20:02:59.6643685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:59.6656461Z 2022-03-17T20:02:59.6656889Z Running tests... 2022-03-17T20:02:59.6657624Z ---------------------------------------------------------------------- 2022-03-17T20:03:00.2591769Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66684 2022-03-17T20:03:00.2624706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66685 2022-03-17T20:03:01.1511471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:01.1512366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:01.1803243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:01.1804999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:01.1805901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:01.1824815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:01.5599832Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:01.6173542Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:03.9715306Z ok (4.306s) 2022-03-17T20:03:03.9715719Z 2022-03-17T20:03:03.9716777Z ---------------------------------------------------------------------- 2022-03-17T20:03:03.9717581Z Ran 1 test in 4.306s 2022-03-17T20:03:03.9717987Z 2022-03-17T20:03:03.9718663Z OK 2022-03-17T20:03:03.9719010Z 2022-03-17T20:03:03.9719323Z Generating XML reports... 2022-03-17T20:03:03.9801777Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200259.xml 2022-03-17T20:03:06.4437857Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:06.4450116Z 2022-03-17T20:03:06.4450498Z Running tests... 2022-03-17T20:03:06.4450986Z ---------------------------------------------------------------------- 2022-03-17T20:03:07.0406576Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66899 2022-03-17T20:03:07.0445541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66900 2022-03-17T20:03:07.9079264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:07.9080495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:07.9670125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:07.9671673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:07.9673288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:07.9693128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:08.3529243Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:08.4044790Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:10.3539541Z ok (3.908s) 2022-03-17T20:03:10.3540045Z 2022-03-17T20:03:10.3541079Z ---------------------------------------------------------------------- 2022-03-17T20:03:10.3541859Z Ran 1 test in 3.909s 2022-03-17T20:03:10.3542272Z 2022-03-17T20:03:10.3542470Z OK 2022-03-17T20:03:10.3542760Z 2022-03-17T20:03:10.3543045Z Generating XML reports... 2022-03-17T20:03:10.3622093Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200306.xml 2022-03-17T20:03:12.8356140Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:12.8369961Z 2022-03-17T20:03:12.8370359Z Running tests... 2022-03-17T20:03:12.8371023Z ---------------------------------------------------------------------- 2022-03-17T20:03:13.4299268Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67114 2022-03-17T20:03:13.4332116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67115 2022-03-17T20:03:14.3452177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:14.3453716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:14.3571415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:14.3572397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:14.3573162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:14.3662486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:16.3411063Z ok (3.503s) 2022-03-17T20:03:16.3411444Z 2022-03-17T20:03:16.3412296Z ---------------------------------------------------------------------- 2022-03-17T20:03:16.3413107Z Ran 1 test in 3.504s 2022-03-17T20:03:16.3413479Z 2022-03-17T20:03:16.3413668Z OK 2022-03-17T20:03:16.3413970Z 2022-03-17T20:03:16.3414229Z Generating XML reports... 2022-03-17T20:03:16.3493691Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200312.xml 2022-03-17T20:03:18.9312964Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:18.9327857Z 2022-03-17T20:03:18.9328154Z Running tests... 2022-03-17T20:03:18.9328724Z ---------------------------------------------------------------------- 2022-03-17T20:03:19.5265971Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67322 2022-03-17T20:03:19.5302695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67323 2022-03-17T20:03:20.4423113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:20.4423704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:20.4539740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:20.4541376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:20.4543575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:20.4631942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:20.8377449Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:20.8899689Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:22.6380649Z ok (3.705s) 2022-03-17T20:03:22.6381233Z 2022-03-17T20:03:22.6382486Z ---------------------------------------------------------------------- 2022-03-17T20:03:22.6383505Z Ran 1 test in 3.705s 2022-03-17T20:03:22.6383928Z 2022-03-17T20:03:22.6384115Z OK 2022-03-17T20:03:22.6384390Z 2022-03-17T20:03:22.6384736Z Generating XML reports... 2022-03-17T20:03:22.6463157Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200318.xml 2022-03-17T20:03:25.1115392Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:25.1128352Z 2022-03-17T20:03:25.1128622Z Running tests... 2022-03-17T20:03:25.1129076Z ---------------------------------------------------------------------- 2022-03-17T20:03:25.7091020Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67537 2022-03-17T20:03:25.7138736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67538 2022-03-17T20:03:26.5827447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:26.5828236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:26.6229954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:26.6232107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:26.6234585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:26.6236467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:27.0939922Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:27.1536784Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:03:29.3217695Z ok (4.208s) 2022-03-17T20:03:29.3218322Z 2022-03-17T20:03:29.3219404Z ---------------------------------------------------------------------- 2022-03-17T20:03:29.3220185Z Ran 1 test in 4.209s 2022-03-17T20:03:29.3220530Z 2022-03-17T20:03:29.3221256Z OK 2022-03-17T20:03:29.3221557Z 2022-03-17T20:03:29.3221849Z Generating XML reports... 2022-03-17T20:03:29.3300802Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200325.xml 2022-03-17T20:03:31.7888130Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:31.7899663Z 2022-03-17T20:03:31.7899997Z Running tests... 2022-03-17T20:03:31.7900463Z ---------------------------------------------------------------------- 2022-03-17T20:03:32.3788679Z test_common_errors (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:32.3789477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:03:32.3883586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:32.3884295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:03:32.3908215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:32.3908913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:03:32.3939060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:32.3939772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:03:32.4019903Z ok (0.612s) 2022-03-17T20:03:32.4020045Z 2022-03-17T20:03:32.4020494Z ---------------------------------------------------------------------- 2022-03-17T20:03:32.4020805Z Ran 1 test in 0.612s 2022-03-17T20:03:32.4020949Z 2022-03-17T20:03:32.4021028Z OK 2022-03-17T20:03:32.4021143Z 2022-03-17T20:03:32.4021262Z Generating XML reports... 2022-03-17T20:03:32.4053529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220317200331.xml 2022-03-17T20:03:34.8559433Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:34.8573091Z 2022-03-17T20:03:34.8573506Z Running tests... 2022-03-17T20:03:34.8574029Z ---------------------------------------------------------------------- 2022-03-17T20:03:35.4421872Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:35.4424194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:03:37.4599792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:37.4601679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:03:38.4795482Z ok (3.622s) 2022-03-17T20:03:38.4795852Z 2022-03-17T20:03:38.4800000Z ---------------------------------------------------------------------- 2022-03-17T20:03:38.4800433Z Ran 1 test in 3.622s 2022-03-17T20:03:38.4800610Z 2022-03-17T20:03:38.4800702Z OK 2022-03-17T20:03:38.4800828Z 2022-03-17T20:03:38.4800970Z Generating XML reports... 2022-03-17T20:03:38.4831268Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220317200334.xml 2022-03-17T20:03:41.6348628Z Running distributed/test_c10d_spawn_gloo ... [2022-03-17 20:03:41.633919] 2022-03-17T20:03:41.6350660Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:03:41.634092] 2022-03-17T20:03:42.5138086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwal8nqo 2022-03-17T20:03:42.5140067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwal8nqo/_remote_module_non_sriptable.py 2022-03-17T20:03:43.0990078Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-03-17T20:03:43.0992042Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-03-17T20:03:43.0993120Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-03-17T20:03:43.0994156Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-03-17T20:03:43.0996321Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-03-17T20:03:43.0998413Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:03:43.0999447Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:03:43.1000448Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:03:43.1001480Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:03:43.1002513Z 2022-03-17T20:03:43.1003254Z 2022-03-17T20:03:43.1006055Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-03-17T20:03:43.1008950Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1009806Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1010600Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1011402Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1012447Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1013220Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1013986Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.1014731Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:03:43.9154314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ds1_iai 2022-03-17T20:03:43.9155638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ds1_iai/_remote_module_non_sriptable.py 2022-03-17T20:03:44.4908671Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:03:44.4919227Z 2022-03-17T20:03:44.4919832Z Running tests... 2022-03-17T20:03:44.4921014Z ---------------------------------------------------------------------- 2022-03-17T20:03:44.5089217Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:03:44.5269891Z ok (0.035s) 2022-03-17T20:03:44.5270201Z 2022-03-17T20:03:44.5270849Z ---------------------------------------------------------------------- 2022-03-17T20:03:44.5271460Z Ran 1 test in 0.035s 2022-03-17T20:03:44.5271760Z 2022-03-17T20:03:44.5271918Z OK 2022-03-17T20:03:44.5272152Z 2022-03-17T20:03:44.5272374Z Generating XML reports... 2022-03-17T20:03:44.5304367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200344.xml 2022-03-17T20:03:47.0005641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl2_1_d_ 2022-03-17T20:03:47.0006269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl2_1_d_/_remote_module_non_sriptable.py 2022-03-17T20:03:47.5726668Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:03:47.5737961Z 2022-03-17T20:03:47.5738441Z Running tests... 2022-03-17T20:03:47.5739649Z ---------------------------------------------------------------------- 2022-03-17T20:03:56.8422208Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:03:56.8654587Z ok (9.291s) 2022-03-17T20:03:56.8660625Z 2022-03-17T20:03:56.8661654Z ---------------------------------------------------------------------- 2022-03-17T20:03:56.8662436Z Ran 1 test in 9.292s 2022-03-17T20:03:56.8662851Z 2022-03-17T20:03:56.8663038Z OK 2022-03-17T20:03:56.8663331Z 2022-03-17T20:03:56.8663609Z Generating XML reports... 2022-03-17T20:03:56.8744941Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200347.xml 2022-03-17T20:04:00.3350291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr38f5qvi 2022-03-17T20:04:00.3351669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr38f5qvi/_remote_module_non_sriptable.py 2022-03-17T20:04:00.9088954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:00.9100138Z 2022-03-17T20:04:00.9100432Z Running tests... 2022-03-17T20:04:00.9101644Z ---------------------------------------------------------------------- 2022-03-17T20:04:15.3571599Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:15.7911279Z ok (14.881s) 2022-03-17T20:04:15.7914272Z 2022-03-17T20:04:15.7915172Z ---------------------------------------------------------------------- 2022-03-17T20:04:15.7915976Z Ran 1 test in 14.882s 2022-03-17T20:04:15.7916358Z 2022-03-17T20:04:15.7916553Z OK 2022-03-17T20:04:15.7916866Z 2022-03-17T20:04:15.7917136Z Generating XML reports... 2022-03-17T20:04:15.7990205Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200400.xml 2022-03-17T20:04:19.4237841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuheb99df 2022-03-17T20:04:19.4239156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuheb99df/_remote_module_non_sriptable.py 2022-03-17T20:04:19.9995246Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:20.0006463Z 2022-03-17T20:04:20.0006752Z Running tests... 2022-03-17T20:04:20.0007884Z ---------------------------------------------------------------------- 2022-03-17T20:04:20.8794349Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62jc60l5 2022-03-17T20:04:20.8795994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62jc60l5/_remote_module_non_sriptable.py 2022-03-17T20:04:20.9260113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv5irwfi 2022-03-17T20:04:20.9261488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv5irwfi/_remote_module_non_sriptable.py 2022-03-17T20:04:23.3159376Z ok (3.315s) 2022-03-17T20:04:23.3159782Z 2022-03-17T20:04:23.3164449Z ---------------------------------------------------------------------- 2022-03-17T20:04:23.3166088Z Ran 1 test in 3.315s 2022-03-17T20:04:23.3166557Z 2022-03-17T20:04:23.3166812Z OK 2022-03-17T20:04:23.3167164Z 2022-03-17T20:04:23.3168133Z Generating XML reports... 2022-03-17T20:04:23.3262427Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200419.xml 2022-03-17T20:04:25.7484236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0o6_ep1 2022-03-17T20:04:25.7485618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0o6_ep1/_remote_module_non_sriptable.py 2022-03-17T20:04:26.3207942Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:26.3216232Z 2022-03-17T20:04:26.3216594Z Running tests... 2022-03-17T20:04:26.3217755Z ---------------------------------------------------------------------- 2022-03-17T20:04:27.7570687Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8tkzuko 2022-03-17T20:04:27.7572334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8tkzuko/_remote_module_non_sriptable.py 2022-03-17T20:04:27.8092752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn71ktvxs 2022-03-17T20:04:27.8093389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn71ktvxs/_remote_module_non_sriptable.py 2022-03-17T20:04:30.8300953Z ok (4.508s) 2022-03-17T20:04:30.8301424Z 2022-03-17T20:04:30.8302498Z ---------------------------------------------------------------------- 2022-03-17T20:04:30.8303398Z Ran 1 test in 4.508s 2022-03-17T20:04:30.8303837Z 2022-03-17T20:04:30.8304079Z OK 2022-03-17T20:04:30.8304416Z 2022-03-17T20:04:30.8304732Z Generating XML reports... 2022-03-17T20:04:30.8399831Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200426.xml 2022-03-17T20:04:33.4024159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ge1qkhx 2022-03-17T20:04:33.4024806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ge1qkhx/_remote_module_non_sriptable.py 2022-03-17T20:04:33.9760457Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:33.9772530Z 2022-03-17T20:04:33.9773100Z Running tests... 2022-03-17T20:04:33.9774142Z ---------------------------------------------------------------------- 2022-03-17T20:04:35.1890483Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot62f2dc 2022-03-17T20:04:35.1892125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot62f2dc/_remote_module_non_sriptable.py 2022-03-17T20:04:35.2674640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01xa7qyg 2022-03-17T20:04:35.2675956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01xa7qyg/_remote_module_non_sriptable.py 2022-03-17T20:04:38.1960440Z ok (4.218s) 2022-03-17T20:04:38.1961277Z 2022-03-17T20:04:38.1963548Z ---------------------------------------------------------------------- 2022-03-17T20:04:38.1964342Z Ran 1 test in 4.219s 2022-03-17T20:04:38.1964713Z 2022-03-17T20:04:38.1966483Z OK 2022-03-17T20:04:38.1966906Z 2022-03-17T20:04:38.1967188Z Generating XML reports... 2022-03-17T20:04:38.2134636Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200433.xml 2022-03-17T20:04:40.7652758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolyxoyje 2022-03-17T20:04:40.7654246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolyxoyje/_remote_module_non_sriptable.py 2022-03-17T20:04:41.3416393Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:41.3425155Z 2022-03-17T20:04:41.3425870Z Running tests... 2022-03-17T20:04:41.3427194Z ---------------------------------------------------------------------- 2022-03-17T20:04:42.7989269Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpde81xxgi 2022-03-17T20:04:42.7990442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpde81xxgi/_remote_module_non_sriptable.py 2022-03-17T20:04:42.8281691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplaj13v4h 2022-03-17T20:04:42.8282569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplaj13v4h/_remote_module_non_sriptable.py 2022-03-17T20:04:45.6889383Z ok (4.346s) 2022-03-17T20:04:45.6891625Z 2022-03-17T20:04:45.6892629Z ---------------------------------------------------------------------- 2022-03-17T20:04:45.6893495Z Ran 1 test in 4.347s 2022-03-17T20:04:45.6893874Z 2022-03-17T20:04:45.6894056Z OK 2022-03-17T20:04:45.6894365Z 2022-03-17T20:04:45.6899326Z Generating XML reports... 2022-03-17T20:04:45.7036549Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200441.xml 2022-03-17T20:04:48.2376923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rnl_0zj 2022-03-17T20:04:48.2378316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rnl_0zj/_remote_module_non_sriptable.py 2022-03-17T20:04:48.8118089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:48.8126302Z 2022-03-17T20:04:48.8126515Z Running tests... 2022-03-17T20:04:48.8127511Z ---------------------------------------------------------------------- 2022-03-17T20:04:48.8345645Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69751 2022-03-17T20:04:48.8393274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69752 2022-03-17T20:04:49.6833588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp799qsjxa 2022-03-17T20:04:49.6834719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp799qsjxa/_remote_module_non_sriptable.py 2022-03-17T20:04:49.7385489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1eiferv 2022-03-17T20:04:49.7386686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1eiferv/_remote_module_non_sriptable.py 2022-03-17T20:04:50.2516205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:50.3157241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:50.3267880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:04:50.3270065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:04:50.3271347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:04:50.3273047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:04:52.7501908Z ok (3.937s) 2022-03-17T20:04:52.7502270Z 2022-03-17T20:04:52.7503437Z ---------------------------------------------------------------------- 2022-03-17T20:04:52.7504248Z Ran 1 test in 3.937s 2022-03-17T20:04:52.7504620Z 2022-03-17T20:04:52.7504804Z OK 2022-03-17T20:04:52.7505097Z 2022-03-17T20:04:52.7505367Z Generating XML reports... 2022-03-17T20:04:52.7589167Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200448.xml 2022-03-17T20:04:55.2917338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmodgply 2022-03-17T20:04:55.2918663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmodgply/_remote_module_non_sriptable.py 2022-03-17T20:04:55.8669710Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:04:55.8680085Z 2022-03-17T20:04:55.8680431Z Running tests... 2022-03-17T20:04:55.8886472Z ---------------------------------------------------------------------- 2022-03-17T20:04:55.8887841Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69965 2022-03-17T20:04:55.8922782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69966 2022-03-17T20:04:56.7223727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqxx5zcb 2022-03-17T20:04:56.7224337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqxx5zcb/_remote_module_non_sriptable.py 2022-03-17T20:04:56.7859954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82yc7346 2022-03-17T20:04:56.7860554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82yc7346/_remote_module_non_sriptable.py 2022-03-17T20:04:57.2951010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:57.3558823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:57.3670576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:04:57.3676205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:04:57.3677840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:04:57.3773508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:04:59.8032650Z ok (3.935s) 2022-03-17T20:04:59.8033185Z 2022-03-17T20:04:59.8034348Z ---------------------------------------------------------------------- 2022-03-17T20:04:59.8035133Z Ran 1 test in 3.935s 2022-03-17T20:04:59.8035487Z 2022-03-17T20:04:59.8035709Z OK 2022-03-17T20:04:59.8035997Z 2022-03-17T20:04:59.8036271Z Generating XML reports... 2022-03-17T20:04:59.8136490Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200455.xml 2022-03-17T20:05:02.3058963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb1r_i8g 2022-03-17T20:05:02.3061085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb1r_i8g/_remote_module_non_sriptable.py 2022-03-17T20:05:02.8821445Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:05:02.8833752Z 2022-03-17T20:05:02.8834464Z Running tests... 2022-03-17T20:05:02.8835843Z ---------------------------------------------------------------------- 2022-03-17T20:05:02.9043640Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70179 2022-03-17T20:05:02.9081251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70180 2022-03-17T20:05:03.7787161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_bg7wi9 2022-03-17T20:05:03.7787798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_bg7wi9/_remote_module_non_sriptable.py 2022-03-17T20:05:03.8072783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptixyo1oa 2022-03-17T20:05:03.8073456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptixyo1oa/_remote_module_non_sriptable.py 2022-03-17T20:05:04.3489559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:04.3853633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:04.4069152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:05:04.4071573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:04.4073376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:05:04.4075115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:06.8163117Z ok (3.933s) 2022-03-17T20:05:06.8163673Z 2022-03-17T20:05:06.8164970Z ---------------------------------------------------------------------- 2022-03-17T20:05:06.8165753Z Ran 1 test in 3.933s 2022-03-17T20:05:06.8166197Z 2022-03-17T20:05:06.8166472Z OK 2022-03-17T20:05:06.8166985Z 2022-03-17T20:05:06.8167257Z Generating XML reports... 2022-03-17T20:05:06.8264745Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200502.xml 2022-03-17T20:05:09.3153292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc5gm5yy 2022-03-17T20:05:09.3154480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc5gm5yy/_remote_module_non_sriptable.py 2022-03-17T20:05:09.8923877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:05:09.8936972Z 2022-03-17T20:05:09.8937413Z Running tests... 2022-03-17T20:05:09.8938409Z ---------------------------------------------------------------------- 2022-03-17T20:05:09.9148573Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70393 2022-03-17T20:05:09.9183848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70394 2022-03-17T20:05:10.7683948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpays8x3uq 2022-03-17T20:05:10.7684531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpays8x3uq/_remote_module_non_sriptable.py 2022-03-17T20:05:10.8171384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllq__mzt 2022-03-17T20:05:10.8171988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllq__mzt/_remote_module_non_sriptable.py 2022-03-17T20:05:11.3370764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:11.3871433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:11.3989107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:05:11.3990899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:05:11.3992799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:11.4092864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:13.6270396Z ok (3.733s) 2022-03-17T20:05:13.6270832Z 2022-03-17T20:05:13.6271822Z ---------------------------------------------------------------------- 2022-03-17T20:05:13.6272653Z Ran 1 test in 3.733s 2022-03-17T20:05:13.6273020Z 2022-03-17T20:05:13.6273213Z OK 2022-03-17T20:05:13.6273499Z 2022-03-17T20:05:13.6273777Z Generating XML reports... 2022-03-17T20:05:13.6373705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200509.xml 2022-03-17T20:05:16.0398546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0_w3ii5 2022-03-17T20:05:16.0399142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0_w3ii5/_remote_module_non_sriptable.py 2022-03-17T20:05:16.6171475Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:05:16.6184764Z 2022-03-17T20:05:16.6185046Z Running tests... 2022-03-17T20:05:16.6185573Z ---------------------------------------------------------------------- 2022-03-17T20:05:16.6392607Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70607 2022-03-17T20:05:16.6439871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70608 2022-03-17T20:05:17.5403665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uyrp56q 2022-03-17T20:05:17.5404553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uyrp56q/_remote_module_non_sriptable.py 2022-03-17T20:05:17.5587625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp04dt7g8 2022-03-17T20:05:17.5588904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp04dt7g8/_remote_module_non_sriptable.py 2022-03-17T20:05:18.1230050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:18.1297137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:18.1440439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:05:18.1443142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:18.1444511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:05:18.1446296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:20.4540760Z ok (3.835s) 2022-03-17T20:05:20.4541321Z 2022-03-17T20:05:20.4542763Z ---------------------------------------------------------------------- 2022-03-17T20:05:20.4543814Z Ran 1 test in 3.836s 2022-03-17T20:05:20.4544181Z 2022-03-17T20:05:20.4544363Z OK 2022-03-17T20:05:20.4544732Z 2022-03-17T20:05:20.4545099Z Generating XML reports... 2022-03-17T20:05:20.4632348Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200516.xml 2022-03-17T20:05:22.9254981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc58c767t 2022-03-17T20:05:22.9256331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc58c767t/_remote_module_non_sriptable.py 2022-03-17T20:05:23.5014405Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:05:23.5027236Z 2022-03-17T20:05:23.5028133Z Running tests... 2022-03-17T20:05:23.5029332Z ---------------------------------------------------------------------- 2022-03-17T20:05:23.5249589Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70821 2022-03-17T20:05:23.5285996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70822 2022-03-17T20:05:24.4408956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rvc47kx 2022-03-17T20:05:24.4410668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rvc47kx/_remote_module_non_sriptable.py 2022-03-17T20:05:24.4556505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsnueglp 2022-03-17T20:05:24.4558208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsnueglp/_remote_module_non_sriptable.py 2022-03-17T20:05:25.0137072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:25.0326417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:25.0538171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:05:25.0539865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:25.0540849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:05:25.0542459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:27.3368320Z ok (3.834s) 2022-03-17T20:05:27.3368826Z 2022-03-17T20:05:27.3370003Z ---------------------------------------------------------------------- 2022-03-17T20:05:27.3370996Z Ran 1 test in 3.834s 2022-03-17T20:05:27.3371473Z 2022-03-17T20:05:27.3371666Z OK 2022-03-17T20:05:27.3371967Z 2022-03-17T20:05:27.3372280Z Generating XML reports... 2022-03-17T20:05:27.3465826Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200523.xml 2022-03-17T20:05:29.7740865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3w0a2_x6 2022-03-17T20:05:29.7741527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3w0a2_x6/_remote_module_non_sriptable.py 2022-03-17T20:05:30.3495164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:05:30.3506279Z 2022-03-17T20:05:30.3506503Z Running tests... 2022-03-17T20:05:30.3506936Z ---------------------------------------------------------------------- 2022-03-17T20:05:30.3718033Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71035 2022-03-17T20:05:30.3751007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71036 2022-03-17T20:05:31.2367941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpska0hmko 2022-03-17T20:05:31.2368587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpska0hmko/_remote_module_non_sriptable.py 2022-03-17T20:05:31.2695948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxr862sjg 2022-03-17T20:05:31.2696719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxr862sjg/_remote_module_non_sriptable.py 2022-03-17T20:05:31.8112734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:31.8543684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:31.8755358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:05:31.8758248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:31.8759647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:05:31.8761399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:34.1855432Z ok (3.834s) 2022-03-17T20:05:34.1856023Z 2022-03-17T20:05:34.1857195Z ---------------------------------------------------------------------- 2022-03-17T20:05:34.1858205Z Ran 1 test in 3.835s 2022-03-17T20:05:34.1858692Z 2022-03-17T20:05:34.1858880Z OK 2022-03-17T20:05:34.1859156Z 2022-03-17T20:05:34.1859459Z Generating XML reports... 2022-03-17T20:05:34.1950195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200530.xml 2022-03-17T20:05:36.5656070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwkks5ue 2022-03-17T20:05:36.5656575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwkks5ue/_remote_module_non_sriptable.py 2022-03-17T20:05:37.1419915Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:05:37.1432549Z 2022-03-17T20:05:37.1432964Z Running tests... 2022-03-17T20:05:37.1433396Z ---------------------------------------------------------------------- 2022-03-17T20:05:37.1658293Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71249 2022-03-17T20:05:37.1686769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71250 2022-03-17T20:05:37.9956038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzdk_e0o 2022-03-17T20:05:37.9956543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzdk_e0o/_remote_module_non_sriptable.py 2022-03-17T20:05:38.0761732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsc27li_ 2022-03-17T20:05:38.0762703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsc27li_/_remote_module_non_sriptable.py 2022-03-17T20:05:38.5679321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:38.6661919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:38.6799666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:05:38.6801131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:05:38.6803477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:38.6805519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:05:40.9796458Z ok (3.836s) 2022-03-17T20:05:40.9796980Z 2022-03-17T20:05:40.9798114Z ---------------------------------------------------------------------- 2022-03-17T20:05:40.9799052Z Ran 1 test in 3.836s 2022-03-17T20:05:40.9799502Z 2022-03-17T20:05:40.9799772Z OK 2022-03-17T20:05:40.9800100Z 2022-03-17T20:05:40.9800383Z Generating XML reports... 2022-03-17T20:05:40.9897696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200537.xml 2022-03-17T20:05:44.2504740Z Running distributed/test_c10d_spawn_nccl ... [2022-03-17 20:05:44.249303] 2022-03-17T20:05:44.2506716Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:05:44.249488] 2022-03-17T20:05:45.1037045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazxfsijk 2022-03-17T20:05:45.1037558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazxfsijk/_remote_module_non_sriptable.py 2022-03-17T20:05:45.6793668Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]> 2022-03-17T20:05:45.6795083Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:05:45.6795750Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:05:45.6796400Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:05:45.6796886Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:05:45.6797303Z 2022-03-17T20:05:45.6797669Z 2022-03-17T20:05:45.6799441Z , <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>]> 2022-03-17T20:05:45.6802497Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:45.6803413Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:45.6804626Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:45.6805559Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:45.6806484Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:45.6807401Z test_reduce (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:45.6808326Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:05:46.5520548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq20ywzvd 2022-03-17T20:05:46.5522349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq20ywzvd/_remote_module_non_sriptable.py 2022-03-17T20:05:47.1239494Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:05:47.1250935Z 2022-03-17T20:05:47.1251632Z Running tests... 2022-03-17T20:05:47.1253031Z ---------------------------------------------------------------------- 2022-03-17T20:05:48.6645979Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynue2rih 2022-03-17T20:05:48.6647586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynue2rih/_remote_module_non_sriptable.py 2022-03-17T20:05:48.6721177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbbh0d2_ 2022-03-17T20:05:48.6723017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbbh0d2_/_remote_module_non_sriptable.py 2022-03-17T20:05:49.3048072Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:05:49.6317053Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:05:51.5995163Z ok (4.474s) 2022-03-17T20:05:51.5998813Z 2022-03-17T20:05:51.6000053Z ---------------------------------------------------------------------- 2022-03-17T20:05:51.6001095Z Ran 1 test in 4.475s 2022-03-17T20:05:51.6001597Z 2022-03-17T20:05:51.6002057Z OK 2022-03-17T20:05:51.6002428Z 2022-03-17T20:05:51.6002840Z Generating XML reports... 2022-03-17T20:05:51.6126089Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200547.xml 2022-03-17T20:05:54.1119311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rsxqk3n 2022-03-17T20:05:54.1121195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rsxqk3n/_remote_module_non_sriptable.py 2022-03-17T20:05:54.6814567Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:05:54.6826076Z 2022-03-17T20:05:54.6826800Z Running tests... 2022-03-17T20:05:54.6828043Z ---------------------------------------------------------------------- 2022-03-17T20:05:55.9304961Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnzlb742 2022-03-17T20:05:55.9305725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnzlb742/_remote_module_non_sriptable.py 2022-03-17T20:05:55.9620822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkr6bjjjk 2022-03-17T20:05:55.9621454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkr6bjjjk/_remote_module_non_sriptable.py 2022-03-17T20:05:56.5462264Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:05:56.6231073Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:05:58.9561206Z ok (4.273s) 2022-03-17T20:05:58.9564300Z 2022-03-17T20:05:58.9565686Z ---------------------------------------------------------------------- 2022-03-17T20:05:58.9566657Z Ran 1 test in 4.274s 2022-03-17T20:05:58.9567085Z 2022-03-17T20:05:58.9567316Z OK 2022-03-17T20:05:58.9567637Z 2022-03-17T20:05:58.9567966Z Generating XML reports... 2022-03-17T20:05:58.9698720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200554.xml 2022-03-17T20:06:01.5047076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwf39uqv 2022-03-17T20:06:01.5048475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwf39uqv/_remote_module_non_sriptable.py 2022-03-17T20:06:02.0740567Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:02.0749050Z 2022-03-17T20:06:02.0749693Z Running tests... 2022-03-17T20:06:02.0750861Z ---------------------------------------------------------------------- 2022-03-17T20:06:03.5519561Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a95owr6 2022-03-17T20:06:03.5520155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a95owr6/_remote_module_non_sriptable.py 2022-03-17T20:06:03.5774797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy85rt_d6 2022-03-17T20:06:03.5775947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy85rt_d6/_remote_module_non_sriptable.py 2022-03-17T20:06:04.1693070Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:04.2250976Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:06.4722309Z ok (4.397s) 2022-03-17T20:06:06.4723441Z 2022-03-17T20:06:06.4725875Z ---------------------------------------------------------------------- 2022-03-17T20:06:06.4726853Z Ran 1 test in 4.397s 2022-03-17T20:06:06.4730322Z 2022-03-17T20:06:06.4730579Z OK 2022-03-17T20:06:06.4731014Z 2022-03-17T20:06:06.4731373Z Generating XML reports... 2022-03-17T20:06:06.4875672Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200602.xml 2022-03-17T20:06:09.0212190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmt5y4xan 2022-03-17T20:06:09.0212755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmt5y4xan/_remote_module_non_sriptable.py 2022-03-17T20:06:09.5927673Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:09.5939048Z 2022-03-17T20:06:09.5939638Z Running tests... 2022-03-17T20:06:09.5940088Z ---------------------------------------------------------------------- 2022-03-17T20:06:10.8323162Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2gmddsj 2022-03-17T20:06:10.8324690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2gmddsj/_remote_module_non_sriptable.py 2022-03-17T20:06:10.8597145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjalzbyik 2022-03-17T20:06:10.8598316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjalzbyik/_remote_module_non_sriptable.py 2022-03-17T20:06:11.4673014Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:11.5249716Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:13.7956786Z ok (4.201s) 2022-03-17T20:06:13.7957020Z 2022-03-17T20:06:13.7958256Z ---------------------------------------------------------------------- 2022-03-17T20:06:13.7960598Z Ran 1 test in 4.202s 2022-03-17T20:06:13.7960978Z 2022-03-17T20:06:13.7961181Z OK 2022-03-17T20:06:13.7961491Z 2022-03-17T20:06:13.7961792Z Generating XML reports... 2022-03-17T20:06:13.8063810Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200609.xml 2022-03-17T20:06:16.1758266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekkmqzc5 2022-03-17T20:06:16.1760026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekkmqzc5/_remote_module_non_sriptable.py 2022-03-17T20:06:16.7455684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:16.7467775Z 2022-03-17T20:06:16.7468452Z Running tests... 2022-03-17T20:06:16.7469621Z ---------------------------------------------------------------------- 2022-03-17T20:06:16.7663788Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72394 2022-03-17T20:06:16.7692540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72395 2022-03-17T20:06:17.6069169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qb8u5op 2022-03-17T20:06:17.6070641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qb8u5op/_remote_module_non_sriptable.py 2022-03-17T20:06:17.6207672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp926olexz 2022-03-17T20:06:17.6208966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp926olexz/_remote_module_non_sriptable.py 2022-03-17T20:06:18.1834498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:18.1835714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:18.2102321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:18.2105168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:18.2106816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:18.2144205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:18.2793486Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:18.6081735Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:20.6775393Z ok (3.930s) 2022-03-17T20:06:20.6775825Z 2022-03-17T20:06:20.6776780Z ---------------------------------------------------------------------- 2022-03-17T20:06:20.6777534Z Ran 1 test in 3.931s 2022-03-17T20:06:20.6777959Z 2022-03-17T20:06:20.6778240Z OK 2022-03-17T20:06:20.6778677Z 2022-03-17T20:06:20.6780474Z Generating XML reports... 2022-03-17T20:06:20.6858277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200616.xml 2022-03-17T20:06:23.0343106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs0g0j_w 2022-03-17T20:06:23.0344465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs0g0j_w/_remote_module_non_sriptable.py 2022-03-17T20:06:23.6200326Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:23.6214302Z 2022-03-17T20:06:23.6215087Z Running tests... 2022-03-17T20:06:23.6216360Z ---------------------------------------------------------------------- 2022-03-17T20:06:23.6475375Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72611 2022-03-17T20:06:23.6524157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72612 2022-03-17T20:06:24.5082882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2z9nt06 2022-03-17T20:06:24.5083501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2z9nt06/_remote_module_non_sriptable.py 2022-03-17T20:06:24.5536289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84es5zit 2022-03-17T20:06:24.5536884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84es5zit/_remote_module_non_sriptable.py 2022-03-17T20:06:25.0824702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:25.0825755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:25.1291724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:25.1293272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:25.1295261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:25.1335130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:25.1938550Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:25.5387096Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:27.4616096Z ok (3.840s) 2022-03-17T20:06:27.4616530Z 2022-03-17T20:06:27.4617720Z ---------------------------------------------------------------------- 2022-03-17T20:06:27.4618776Z Ran 1 test in 3.840s 2022-03-17T20:06:27.4619258Z 2022-03-17T20:06:27.4619579Z OK 2022-03-17T20:06:27.4619936Z 2022-03-17T20:06:27.4620228Z Generating XML reports... 2022-03-17T20:06:27.4702167Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200623.xml 2022-03-17T20:06:29.9027087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08wsfq4w 2022-03-17T20:06:29.9032969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08wsfq4w/_remote_module_non_sriptable.py 2022-03-17T20:06:30.4726989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:30.4736992Z 2022-03-17T20:06:30.4737302Z Running tests... 2022-03-17T20:06:30.4737748Z ---------------------------------------------------------------------- 2022-03-17T20:06:30.4935529Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72828 2022-03-17T20:06:30.4976156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72829 2022-03-17T20:06:31.3665419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lg5sguh 2022-03-17T20:06:31.3667040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lg5sguh/_remote_module_non_sriptable.py 2022-03-17T20:06:31.3740405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1p9wwvs 2022-03-17T20:06:31.3741911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1p9wwvs/_remote_module_non_sriptable.py 2022-03-17T20:06:31.9369971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:31.9371497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:31.9626370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:31.9627973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:31.9630068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:31.9677627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:32.0632726Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:32.3884463Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:34.3080258Z ok (3.834s) 2022-03-17T20:06:34.3080714Z 2022-03-17T20:06:34.3082126Z ---------------------------------------------------------------------- 2022-03-17T20:06:34.3083094Z Ran 1 test in 3.834s 2022-03-17T20:06:34.3083558Z 2022-03-17T20:06:34.3083768Z OK 2022-03-17T20:06:34.3084071Z 2022-03-17T20:06:34.3084345Z Generating XML reports... 2022-03-17T20:06:34.3165289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200630.xml 2022-03-17T20:06:36.6682125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2ee3d3r 2022-03-17T20:06:36.6682637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2ee3d3r/_remote_module_non_sriptable.py 2022-03-17T20:06:37.2391687Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:37.2404142Z 2022-03-17T20:06:37.2404513Z Running tests... 2022-03-17T20:06:37.2404961Z ---------------------------------------------------------------------- 2022-03-17T20:06:37.2599092Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73045 2022-03-17T20:06:37.2633690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73046 2022-03-17T20:06:38.1499752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfcynas9 2022-03-17T20:06:38.1500569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfcynas9/_remote_module_non_sriptable.py 2022-03-17T20:06:38.1668294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeibv08wt 2022-03-17T20:06:38.1669108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeibv08wt/_remote_module_non_sriptable.py 2022-03-17T20:06:38.7207712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:38.7208871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:38.7431426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:38.7432957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:38.7434907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:38.7517203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:38.8145276Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:39.1389268Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:41.1740681Z ok (3.933s) 2022-03-17T20:06:41.1741979Z 2022-03-17T20:06:41.1743166Z ---------------------------------------------------------------------- 2022-03-17T20:06:41.1743941Z Ran 1 test in 3.933s 2022-03-17T20:06:41.1744305Z 2022-03-17T20:06:41.1744489Z OK 2022-03-17T20:06:41.1744805Z 2022-03-17T20:06:41.1745101Z Generating XML reports... 2022-03-17T20:06:41.1822998Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200637.xml 2022-03-17T20:06:43.6307861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsn619err 2022-03-17T20:06:43.6308512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsn619err/_remote_module_non_sriptable.py 2022-03-17T20:06:44.2208298Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:44.2222281Z 2022-03-17T20:06:44.2222458Z Running tests... 2022-03-17T20:06:44.2223049Z ---------------------------------------------------------------------- 2022-03-17T20:06:44.2440060Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73262 2022-03-17T20:06:44.2479846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73263 2022-03-17T20:06:45.1723898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcasmgu5c 2022-03-17T20:06:45.1724524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcasmgu5c/_remote_module_non_sriptable.py 2022-03-17T20:06:45.2173627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperpswvqd 2022-03-17T20:06:45.2174272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperpswvqd/_remote_module_non_sriptable.py 2022-03-17T20:06:45.7448715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:45.7450125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:45.7978523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:45.7980558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:45.7982524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:45.8060313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:45.8678811Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:46.1978873Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:48.1620468Z ok (3.939s) 2022-03-17T20:06:48.1621066Z 2022-03-17T20:06:48.1622201Z ---------------------------------------------------------------------- 2022-03-17T20:06:48.1623206Z Ran 1 test in 3.940s 2022-03-17T20:06:48.1623700Z 2022-03-17T20:06:48.1623933Z OK 2022-03-17T20:06:48.1624211Z 2022-03-17T20:06:48.1624498Z Generating XML reports... 2022-03-17T20:06:48.1706621Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200644.xml 2022-03-17T20:06:50.5350749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsudtw66 2022-03-17T20:06:50.5351260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsudtw66/_remote_module_non_sriptable.py 2022-03-17T20:06:51.1177684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:51.1192923Z 2022-03-17T20:06:51.1193235Z Running tests... 2022-03-17T20:06:51.1193661Z ---------------------------------------------------------------------- 2022-03-17T20:06:51.1402402Z test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73479 2022-03-17T20:06:51.1442167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73480 2022-03-17T20:06:52.0356705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54uec_6l 2022-03-17T20:06:52.0357814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54uec_6l/_remote_module_non_sriptable.py 2022-03-17T20:06:52.0510498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6asi3jt 2022-03-17T20:06:52.0511009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6asi3jt/_remote_module_non_sriptable.py 2022-03-17T20:06:52.6225145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:52.6225698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:52.6456599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:52.6457873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:52.6459748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:52.6535198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:52.7117628Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:53.0565959Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:55.0537160Z ok (3.934s) 2022-03-17T20:06:55.0537610Z 2022-03-17T20:06:55.0538951Z ---------------------------------------------------------------------- 2022-03-17T20:06:55.0539732Z Ran 1 test in 3.934s 2022-03-17T20:06:55.0540106Z 2022-03-17T20:06:55.0540294Z OK 2022-03-17T20:06:55.0540589Z 2022-03-17T20:06:55.0540873Z Generating XML reports... 2022-03-17T20:06:55.0620153Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200651.xml 2022-03-17T20:06:57.4863520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz19e9r12 2022-03-17T20:06:57.4864023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz19e9r12/_remote_module_non_sriptable.py 2022-03-17T20:06:58.0603678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:06:58.0616596Z 2022-03-17T20:06:58.0616990Z Running tests... 2022-03-17T20:06:58.0617970Z ---------------------------------------------------------------------- 2022-03-17T20:06:58.0834141Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73696 2022-03-17T20:06:58.0864167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73697 2022-03-17T20:06:59.0171189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpben0z4kj 2022-03-17T20:06:59.0171691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpben0z4kj/_remote_module_non_sriptable.py 2022-03-17T20:06:59.0211343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q6i5kbv 2022-03-17T20:06:59.0211858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q6i5kbv/_remote_module_non_sriptable.py 2022-03-17T20:06:59.5895429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:59.5896763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:59.5924039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:59.5925306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:59.5927973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:59.5998443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:59.6645963Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:06:59.9859288Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:07:01.8970500Z ok (3.835s) 2022-03-17T20:07:01.8971324Z 2022-03-17T20:07:01.8972847Z ---------------------------------------------------------------------- 2022-03-17T20:07:01.8973883Z Ran 1 test in 3.835s 2022-03-17T20:07:01.8974256Z 2022-03-17T20:07:01.8974468Z OK 2022-03-17T20:07:01.8974779Z 2022-03-17T20:07:01.8975040Z Generating XML reports... 2022-03-17T20:07:01.9055466Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200658.xml 2022-03-17T20:07:04.9793169Z Running distributed/test_data_parallel ... [2022-03-17 20:07:04.978400] 2022-03-17T20:07:04.9794961Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:07:04.978565] 2022-03-17T20:07:06.4115133Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel 2022-03-17T20:07:06.4135245Z 2022-03-17T20:07:06.4135766Z Running tests... 2022-03-17T20:07:06.4136901Z ---------------------------------------------------------------------- 2022-03-17T20:07:06.8202026Z test_autocast (__main__.TestDataParallel) ... libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:07:16.7253430Z ok (10.311s) 2022-03-17T20:07:17.1168445Z test_data_parallel (__main__.TestDataParallel) ... ok (0.391s) 2022-03-17T20:07:17.3347688Z test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.218s) 2022-03-17T20:07:17.7617500Z test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.427s) 2022-03-17T20:07:17.7684682Z test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.007s) 2022-03-17T20:07:17.7864740Z test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.018s) 2022-03-17T20:07:17.7890620Z test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.7/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-03-17T20:07:17.7892735Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-03-17T20:07:17.7897745Z ok (0.003s) 2022-03-17T20:07:17.7948764Z test_data_parallel_model_device (__main__.TestDataParallel) 2022-03-17T20:07:17.9390879Z Test device[0] check at forward time. ... ok (0.149s) 2022-03-17T20:07:18.0387495Z test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.100s) 2022-03-17T20:07:18.0446430Z test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:07:18.0523134Z test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/comm.py:232: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu". 2022-03-17T20:07:18.0525080Z 'Using -1 to represent CPU tensor is deprecated. Please use a ' 2022-03-17T20:07:18.0733203Z ok (0.029s) 2022-03-17T20:07:18.1154941Z test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.042s) 2022-03-17T20:07:18.1681081Z test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.053s) 2022-03-17T20:07:18.1736858Z test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.005s) 2022-03-17T20:07:18.4237560Z test_data_parallel_rnn (__main__.TestDataParallel) ... ok (0.250s) 2022-03-17T20:07:18.4274821Z test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.004s) 2022-03-17T20:07:18.7883017Z test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.361s) 2022-03-17T20:07:18.8890909Z test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. 2022-03-17T20:07:18.8892757Z warnings.warn('Was asked to gather along dimension 0, but all ' 2022-03-17T20:07:18.9117430Z ok (0.123s) 2022-03-17T20:07:18.9128068Z test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s) 2022-03-17T20:07:18.9671431Z test_gather_gpu (__main__.TestDataParallel) ... ok (0.054s) 2022-03-17T20:07:18.9757359Z test_parallel_apply (__main__.TestDataParallel) ... ok (0.008s) 2022-03-17T20:07:18.9820900Z test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:07:18.9844600Z test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s) 2022-03-17T20:07:18.9956661Z test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.011s) 2022-03-17T20:07:19.0027217Z test_replicate (__main__.TestDataParallel) ... ok (0.007s) 2022-03-17T20:07:19.0088412Z test_replicate_buffers (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:07:19.0146099Z test_save_replica_module (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:07:19.0383446Z test_scatter_cpu (__main__.TestDataParallel) ... ok (0.024s) 2022-03-17T20:07:19.0585254Z test_scatter_gpu (__main__.TestDataParallel) ... ok (0.020s) 2022-03-17T20:07:19.0680793Z test_strided_grad_layout (__main__.TestDataParallel) ... [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:07:24.7861537Z ok (5.727s) 2022-03-17T20:07:24.8021851Z test_zero_grad (__main__.TestDataParallel) ... ok (0.016s) 2022-03-17T20:07:24.8049072Z test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... skip (0.003s) 2022-03-17T20:07:24.8135012Z test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.008s) 2022-03-17T20:07:24.8143326Z test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... skip (0.001s) 2022-03-17T20:07:24.8580460Z test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.043s) 2022-03-17T20:07:24.8945093Z test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.036s) 2022-03-17T20:07:24.9304958Z test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.036s) 2022-03-17T20:07:24.9671905Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.037s) 2022-03-17T20:07:25.0040592Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.037s) 2022-03-17T20:07:25.0406331Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.036s) 2022-03-17T20:07:25.0774039Z test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.037s) 2022-03-17T20:07:25.0789856Z test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip (0.002s) 2022-03-17T20:07:25.1151433Z test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.036s) 2022-03-17T20:07:25.1521721Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.037s) 2022-03-17T20:07:25.1889603Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.037s) 2022-03-17T20:07:25.2252995Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.036s) 2022-03-17T20:07:25.2253812Z 2022-03-17T20:07:25.2255036Z ---------------------------------------------------------------------- 2022-03-17T20:07:25.2255829Z Ran 46 tests in 18.812s 2022-03-17T20:07:25.2256202Z 2022-03-17T20:07:25.2256439Z OK (skipped=3) 2022-03-17T20:07:25.2256784Z 2022-03-17T20:07:25.2266025Z Generating XML reports... 2022-03-17T20:07:25.2326555Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220317200706.xml 2022-03-17T20:07:25.2346538Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20220317200706.xml 2022-03-17T20:07:28.3273717Z Running distributed/test_distributed_spawn ... [2022-03-17 20:07:28.326516] 2022-03-17T20:07:28.3407508Z MPI not available -- MPI backend tests will be skipped 2022-03-17T20:07:28.3415614Z Running distributed tests for the test backend with env init_method 2022-03-17T20:07:28.3417747Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:07:28.341389] 2022-03-17T20:07:29.4742575Z 2022-03-17T20:07:31.5716159Z Running distributed tests for the test backend with file init_method 2022-03-17T20:07:31.5718430Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:07:31.571171] 2022-03-17T20:07:32.6580647Z 2022-03-17T20:07:34.2277407Z Running distributed tests for the nccl backend with env init_method 2022-03-17T20:07:34.2279616Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:07:34.227278] 2022-03-17T20:07:35.3421599Z 2022-03-17T20:07:35.3522326Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T20:07:35.3561311Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3561880Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3562281Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3562655Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3563060Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3563478Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3563929Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3564386Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3564852Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3565356Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3565871Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3566358Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3566851Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3567320Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3567759Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3568251Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3568684Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3569077Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3569463Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3569877Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3570315Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3570750Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3571128Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3571480Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3571869Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3572255Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3572635Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3573025Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3573395Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3573749Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3574102Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3574475Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3574830Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3575175Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3575545Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3575921Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3576307Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3576695Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3577088Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3577501Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3577957Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3578359Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3578753Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3579142Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3579542Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3579911Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3580315Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3580718Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3581082Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3581456Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3581837Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3582223Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3582596Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3582965Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3583347Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3583703Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3584151Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3584510Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3584874Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3585224Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3585557Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3585914Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3586278Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3586653Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3587017Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3587361Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3587713Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3588068Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3588437Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3588807Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3589175Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3589530Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3589867Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3590217Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3590571Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3590927Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3591296Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3591647Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3592000Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3592365Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3592757Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3593156Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3593558Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3594026Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3594435Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3594847Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3595255Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3595642Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3596048Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3596460Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3596869Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3597289Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3597705Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3598121Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3598536Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3598918Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3599278Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3599621Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3600006Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3600330Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3600679Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3601039Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3601384Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3601741Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3602172Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3602543Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3602911Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3603271Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3603647Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3604033Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3604424Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3604806Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3605183Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3605565Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3605939Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3606318Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3606671Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3607001Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3607356Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3607708Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3608070Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3608429Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3608848Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3609326Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3609741Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3610172Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3610555Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3610945Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3611364Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3611790Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3612205Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3612597Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3612999Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3613375Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3613712Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3614075Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3614434Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3614801Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3615184Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3615578Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3615982Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3616409Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3616831Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3617300Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3617816Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3618398Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3618979Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3619545Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3620115Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3620688Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3621248Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3621815Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3622323Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3622774Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3623189Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3623535Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3623901Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3624274Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3624631Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3625066Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3625462Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3625885Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3626315Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3626688Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3627048Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3627424Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3627824Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3628210Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3628593Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3628977Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3629365Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3629764Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3630139Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3630524Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3630908Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3631329Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3631698Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3632077Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3632498Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3632896Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3633240Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3633604Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3633983Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3634363Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3634711Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3635037Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3635385Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3635722Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3636071Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3636411Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3636761Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3637119Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3637448Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3637773Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3638114Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3638473Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3638830Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3639160Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3639478Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3639818Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3640186Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3640567Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3641026Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3641444Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3641819Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3642353Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3642759Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3643148Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3643541Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3643921Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3644303Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3644681Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3645048Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3645412Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3645769Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3646201Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3646651Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3647074Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3647541Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3647946Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3648355Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3648748Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3649127Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3649510Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3649880Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3650239Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3650603Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3650956Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3651307Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3651642Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3651995Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3652338Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3652675Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3652999Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3653335Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3653681Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3654005Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3654342Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3654690Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3655034Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3655365Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3655685Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3656028Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3656365Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3656699Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3657053Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3657440Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3657788Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3658114Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3658456Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3658833Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3659232Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3659625Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3659974Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3660339Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3660721Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3661093Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3661452Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3661813Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3662210Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3662582Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3662937Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3663352Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3663700Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3664043Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3664421Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3664825Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:35.3665230Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:07:36.3925064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:07:36.3935509Z 2022-03-17T20:07:36.3936350Z Running tests... 2022-03-17T20:07:36.3937620Z ---------------------------------------------------------------------- 2022-03-17T20:07:36.9919552Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76128 2022-03-17T20:07:36.9953956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76129 2022-03-17T20:07:36.9955504Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:07:38.0678305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:38.1280251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:38.1283001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:38.1285126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:38.1291963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:38.2294587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:38.5253483Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:38.5256124Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:38.6277723Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:38.6280350Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:38.7677217Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:07:38.9881428Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:07:39.0768456Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:39.0771140Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:39.0772793Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:39.0774947Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:39.0867198Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:39.0869877Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:39.0871911Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:39.0874035Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:39.0967952Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:39.0970637Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:39.0972276Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:07:39.0974403Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:07:40.8058536Z ok (4.412s) 2022-03-17T20:07:40.8059028Z 2022-03-17T20:07:40.8060097Z ---------------------------------------------------------------------- 2022-03-17T20:07:40.8061058Z Ran 1 test in 4.412s 2022-03-17T20:07:40.8061531Z 2022-03-17T20:07:40.8061716Z OK 2022-03-17T20:07:40.8062002Z 2022-03-17T20:07:40.8062298Z Generating XML reports... 2022-03-17T20:07:40.8139979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200736.xml 2022-03-17T20:07:43.4031448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:07:43.4042636Z 2022-03-17T20:07:43.4043405Z Running tests... 2022-03-17T20:07:43.4044895Z ---------------------------------------------------------------------- 2022-03-17T20:07:43.4080023Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:07:43.4080928Z 2022-03-17T20:07:43.4082098Z ---------------------------------------------------------------------- 2022-03-17T20:07:43.4082889Z Ran 1 test in 0.004s 2022-03-17T20:07:43.4083253Z 2022-03-17T20:07:43.4083480Z OK (skipped=1) 2022-03-17T20:07:43.4083810Z 2022-03-17T20:07:43.4084083Z Generating XML reports... 2022-03-17T20:07:43.4118744Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200743.xml 2022-03-17T20:07:45.9313952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:07:45.9326810Z 2022-03-17T20:07:45.9327084Z Running tests... 2022-03-17T20:07:45.9327639Z ---------------------------------------------------------------------- 2022-03-17T20:07:46.5489845Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76410 2022-03-17T20:07:46.5547250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76411 2022-03-17T20:07:46.5548943Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:07:47.6719952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:47.6762160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:47.6763132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:47.6819804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:47.6822179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:47.7774520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:49.5643160Z ok (3.631s) 2022-03-17T20:07:49.5644347Z 2022-03-17T20:07:49.5645454Z ---------------------------------------------------------------------- 2022-03-17T20:07:49.5646465Z Ran 1 test in 3.631s 2022-03-17T20:07:49.5646934Z 2022-03-17T20:07:49.5647153Z OK 2022-03-17T20:07:49.5647564Z 2022-03-17T20:07:49.5647861Z Generating XML reports... 2022-03-17T20:07:49.5727225Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200745.xml 2022-03-17T20:07:52.1631678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:07:52.1643858Z 2022-03-17T20:07:52.1644466Z Running tests... 2022-03-17T20:07:52.1645662Z ---------------------------------------------------------------------- 2022-03-17T20:07:52.7917270Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76618 2022-03-17T20:07:52.7955549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76619 2022-03-17T20:07:52.7956887Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:07:53.9275938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:53.9390110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:53.9390971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:53.9480039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:53.9484227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:54.0402869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:54.3350248Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:07:54.4214702Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:03.8788707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:03.8789987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:03.9026833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:03.9027778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:03.9135503Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:03.9137084Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:03.9145445Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:03.9389502Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:03.9390135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:03.9390719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.0853850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.0864260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.1061261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.1076943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.2333588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.2334571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.2544429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.2551999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.3891322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.3892455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.5222863Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:08:04.5225389Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:08:04.5235875Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:08:04.5238288Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:08:04.5412424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.5413021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.6962304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.6962903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.8494713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.8495395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.9928502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:04.9929139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.0155162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.0160316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.1502698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.1516282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.2966511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.2967142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.3174664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.3175661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.4494484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.4495146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.4687296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.4695860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.6045746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.6046778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.7605055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.7606078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.9155935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:05.9156619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:06.0650992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:06.0662143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:08.9276036Z ok (16.763s) 2022-03-17T20:08:08.9276538Z 2022-03-17T20:08:08.9277622Z ---------------------------------------------------------------------- 2022-03-17T20:08:08.9278475Z Ran 1 test in 16.763s 2022-03-17T20:08:08.9278837Z 2022-03-17T20:08:08.9279011Z OK 2022-03-17T20:08:08.9279308Z 2022-03-17T20:08:08.9279584Z Generating XML reports... 2022-03-17T20:08:08.9344788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200752.xml 2022-03-17T20:08:11.6852270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:11.6864025Z 2022-03-17T20:08:11.6864359Z Running tests... 2022-03-17T20:08:11.6864790Z ---------------------------------------------------------------------- 2022-03-17T20:08:11.6878221Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:08:11.6878514Z 2022-03-17T20:08:11.6878915Z ---------------------------------------------------------------------- 2022-03-17T20:08:11.6879280Z Ran 1 test in 0.002s 2022-03-17T20:08:11.6882265Z 2022-03-17T20:08:11.6884938Z OK (skipped=1) 2022-03-17T20:08:11.6885359Z 2022-03-17T20:08:11.6885629Z Generating XML reports... 2022-03-17T20:08:11.6919345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200811.xml 2022-03-17T20:08:14.2076464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:14.2089421Z 2022-03-17T20:08:14.2089845Z Running tests... 2022-03-17T20:08:14.2090283Z ---------------------------------------------------------------------- 2022-03-17T20:08:14.2108396Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:08:14.2108691Z 2022-03-17T20:08:14.2109228Z ---------------------------------------------------------------------- 2022-03-17T20:08:14.2109606Z Ran 1 test in 0.002s 2022-03-17T20:08:14.2109822Z 2022-03-17T20:08:14.2109927Z OK (skipped=1) 2022-03-17T20:08:14.2110059Z 2022-03-17T20:08:14.2110159Z Generating XML reports... 2022-03-17T20:08:14.2151473Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200814.xml 2022-03-17T20:08:16.8818550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:16.8832787Z 2022-03-17T20:08:16.8833097Z Running tests... 2022-03-17T20:08:16.8833530Z ---------------------------------------------------------------------- 2022-03-17T20:08:17.5199200Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76971 2022-03-17T20:08:17.5244099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76972 2022-03-17T20:08:17.5245435Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:18.6341765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:18.6706324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:18.6708602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:18.6750957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:18.6763301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:18.7718228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:19.0705414Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:19.1559480Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:29.7422780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.7424290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.7698134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.7700877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.7801669Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:29.7804231Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:29.7866507Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:29.7868596Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:29.9041760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.9043531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.9296624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:29.9306398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:30.0636103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:30.0637631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:30.0870940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:30.0872491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:32.9563115Z ok (16.073s) 2022-03-17T20:08:32.9563886Z 2022-03-17T20:08:32.9565604Z ---------------------------------------------------------------------- 2022-03-17T20:08:32.9566763Z Ran 1 test in 16.073s 2022-03-17T20:08:32.9567217Z 2022-03-17T20:08:32.9567465Z OK 2022-03-17T20:08:32.9567849Z 2022-03-17T20:08:32.9568279Z Generating XML reports... 2022-03-17T20:08:32.9650846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200816.xml 2022-03-17T20:08:35.5849432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:35.5862707Z 2022-03-17T20:08:35.5863158Z Running tests... 2022-03-17T20:08:35.5863760Z ---------------------------------------------------------------------- 2022-03-17T20:08:36.1852812Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77473 2022-03-17T20:08:36.1940511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77474 2022-03-17T20:08:36.1942404Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:37.3128639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:37.3167900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:37.3168700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:37.3227666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:37.3229936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:37.4183435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:37.8003448Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:37.8534425Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:37.9459988Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:37.9462674Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:38.2003994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:38.2315921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:38.2471259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:38.2471839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:38.2528107Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:38.2529031Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:38.2536902Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:38.2537810Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:39.9063770Z ok (4.319s) 2022-03-17T20:08:39.9064302Z 2022-03-17T20:08:39.9065475Z ---------------------------------------------------------------------- 2022-03-17T20:08:39.9066495Z Ran 1 test in 4.320s 2022-03-17T20:08:39.9066847Z 2022-03-17T20:08:39.9067046Z OK 2022-03-17T20:08:39.9067334Z 2022-03-17T20:08:39.9067610Z Generating XML reports... 2022-03-17T20:08:39.9143019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200835.xml 2022-03-17T20:08:42.7364917Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:42.7374008Z 2022-03-17T20:08:42.7374352Z Running tests... 2022-03-17T20:08:42.7374905Z ---------------------------------------------------------------------- 2022-03-17T20:08:43.3623765Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77692 2022-03-17T20:08:43.3666487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77693 2022-03-17T20:08:43.3668308Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:44.4471172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:44.4472109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:44.4472698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:44.4473460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:44.4478668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:44.4479193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:44.8315328Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:44.8808342Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:45.2356711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:45.2360304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:45.2514600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:45.2515253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:45.2571719Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:45.2573789Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:45.2576559Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:08:45.2578897Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:08:46.9761230Z ok (4.238s) 2022-03-17T20:08:46.9761635Z 2022-03-17T20:08:46.9763071Z ---------------------------------------------------------------------- 2022-03-17T20:08:46.9764013Z Ran 1 test in 4.238s 2022-03-17T20:08:46.9764441Z 2022-03-17T20:08:46.9769914Z OK 2022-03-17T20:08:46.9770680Z 2022-03-17T20:08:46.9771198Z Generating XML reports... 2022-03-17T20:08:46.9846157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200842.xml 2022-03-17T20:08:49.6104417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:49.6118311Z 2022-03-17T20:08:49.6118633Z Running tests... 2022-03-17T20:08:49.6119062Z ---------------------------------------------------------------------- 2022-03-17T20:08:50.2392839Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77911 2022-03-17T20:08:50.2438152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77912 2022-03-17T20:08:50.2439483Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:51.3620898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:51.3745938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:51.3747933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:51.3824692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:51.3832682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:51.4760087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:51.8562983Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:51.9100506Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:52.1237294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:52.1237923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:54.0557118Z ok (4.443s) 2022-03-17T20:08:54.0557556Z 2022-03-17T20:08:54.0558755Z ---------------------------------------------------------------------- 2022-03-17T20:08:54.0559600Z Ran 1 test in 4.444s 2022-03-17T20:08:54.0559998Z 2022-03-17T20:08:54.0560214Z OK 2022-03-17T20:08:54.0560536Z 2022-03-17T20:08:54.0560827Z Generating XML reports... 2022-03-17T20:08:54.0641641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200849.xml 2022-03-17T20:08:56.7522160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:08:56.7528644Z 2022-03-17T20:08:56.7529151Z Running tests... 2022-03-17T20:08:56.7530331Z ---------------------------------------------------------------------- 2022-03-17T20:08:57.3678455Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78130 2022-03-17T20:08:57.3732139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78131 2022-03-17T20:08:57.3733446Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:08:58.4949494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:58.5352517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:58.5354452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:58.5356208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:58.5363256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:58.6363671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:59.0248991Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:08:59.0777292Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:09:09.6355459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:09.6356793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:09.6618429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:09.6619735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:09.6738452Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:09:09.6741002Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:09:09.6755671Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:09:09.6757791Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:09:12.4111440Z ok (15.657s) 2022-03-17T20:09:12.4112005Z 2022-03-17T20:09:12.4113177Z ---------------------------------------------------------------------- 2022-03-17T20:09:12.4114174Z Ran 1 test in 15.658s 2022-03-17T20:09:12.4114600Z 2022-03-17T20:09:12.4114827Z OK 2022-03-17T20:09:12.4115150Z 2022-03-17T20:09:12.4115420Z Generating XML reports... 2022-03-17T20:09:12.4185328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200856.xml 2022-03-17T20:09:15.1788263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:09:15.1799749Z 2022-03-17T20:09:15.1800500Z Running tests... 2022-03-17T20:09:15.1803272Z ---------------------------------------------------------------------- 2022-03-17T20:09:15.7766551Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78621 2022-03-17T20:09:15.7802862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78622 2022-03-17T20:09:15.7804235Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:16.8302825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:16.8993146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:16.8995359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:16.9010346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:16.9011780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:17.0005860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:17.3870737Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:09:17.4339990Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:09:26.7380776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:26.7421762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:26.7665261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:26.7667041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:26.7732050Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:09:26.7733688Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:09:26.7745308Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:09:26.7747244Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:09:29.5136996Z ok (14.333s) 2022-03-17T20:09:29.5137512Z 2022-03-17T20:09:29.5138688Z ---------------------------------------------------------------------- 2022-03-17T20:09:29.5139701Z Ran 1 test in 14.333s 2022-03-17T20:09:29.5140185Z 2022-03-17T20:09:29.5140382Z OK 2022-03-17T20:09:29.5140685Z 2022-03-17T20:09:29.5140963Z Generating XML reports... 2022-03-17T20:09:29.5217356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200915.xml 2022-03-17T20:09:32.3373814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:09:32.3385480Z 2022-03-17T20:09:32.3385818Z Running tests... 2022-03-17T20:09:32.3386254Z ---------------------------------------------------------------------- 2022-03-17T20:09:32.9384796Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78840 2022-03-17T20:09:32.9412886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78841 2022-03-17T20:09:32.9414221Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:34.0007786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:34.0612289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:34.0614190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:34.0615725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:34.0624296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:34.1627109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:34.5555438Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:09:34.6091009Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:09:34.7046985Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:34.7079049Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:34.9786944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:34.9788169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:34.9943212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:34.9944402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:09:35.0000087Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:09:35.0002275Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:09:35.0005050Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:09:35.0007426Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:09:36.6510580Z ok (4.312s) 2022-03-17T20:09:36.6510975Z 2022-03-17T20:09:36.6512191Z ---------------------------------------------------------------------- 2022-03-17T20:09:36.6513170Z Ran 1 test in 4.312s 2022-03-17T20:09:36.6513570Z 2022-03-17T20:09:36.6513765Z OK 2022-03-17T20:09:36.6514069Z 2022-03-17T20:09:36.6514362Z Generating XML reports... 2022-03-17T20:09:36.6586911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200932.xml 2022-03-17T20:09:39.3043434Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:09:39.3055547Z 2022-03-17T20:09:39.3055857Z Running tests... 2022-03-17T20:09:39.3056293Z ---------------------------------------------------------------------- 2022-03-17T20:09:39.9045462Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79059 2022-03-17T20:09:39.9083737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79060 2022-03-17T20:09:39.9085110Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:40.9708922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:41.0472071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:41.0473985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:41.0517359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:41.0518698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:41.1484760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:42.8177790Z skip (3.511s) 2022-03-17T20:09:42.8178204Z 2022-03-17T20:09:42.8180158Z ---------------------------------------------------------------------- 2022-03-17T20:09:42.8181076Z Ran 1 test in 3.512s 2022-03-17T20:09:42.8181444Z 2022-03-17T20:09:42.8181660Z OK (skipped=1) 2022-03-17T20:09:42.8182015Z 2022-03-17T20:09:42.8182281Z Generating XML reports... 2022-03-17T20:09:42.8259246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200939.xml 2022-03-17T20:09:45.5862498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:09:45.5875262Z 2022-03-17T20:09:45.5876002Z Running tests... 2022-03-17T20:09:45.5877493Z ---------------------------------------------------------------------- 2022-03-17T20:09:46.1928317Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79267 2022-03-17T20:09:46.1975769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79268 2022-03-17T20:09:46.1977176Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:47.2804771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:47.3182581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:47.3183558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:47.3212842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:47.3224819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:47.4189626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:49.3065309Z ok (3.718s) 2022-03-17T20:09:49.3065788Z 2022-03-17T20:09:49.3067131Z ---------------------------------------------------------------------- 2022-03-17T20:09:49.3068179Z Ran 1 test in 3.719s 2022-03-17T20:09:49.3068645Z 2022-03-17T20:09:49.3068889Z OK 2022-03-17T20:09:49.3069279Z 2022-03-17T20:09:49.3069636Z Generating XML reports... 2022-03-17T20:09:49.3141991Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200945.xml 2022-03-17T20:09:51.9022940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:09:51.9034878Z 2022-03-17T20:09:51.9035172Z Running tests... 2022-03-17T20:09:51.9035730Z ---------------------------------------------------------------------- 2022-03-17T20:09:52.5402672Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79475 2022-03-17T20:09:52.5447988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79476 2022-03-17T20:09:52.5449984Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:09:53.6982527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:53.7114733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:53.7116751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:53.7183530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:53.7194399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:53.8129375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:54.1966784Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:09:54.2564888Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:03.6706977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:10:03.6927408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:10:03.7625521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:10:03.7626163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:10:06.5797442Z ok (14.675s) 2022-03-17T20:10:06.5797858Z 2022-03-17T20:10:06.5798875Z ---------------------------------------------------------------------- 2022-03-17T20:10:06.5799693Z Ran 1 test in 14.676s 2022-03-17T20:10:06.5800047Z 2022-03-17T20:10:06.5804317Z OK 2022-03-17T20:10:06.5804806Z 2022-03-17T20:10:06.5805189Z Generating XML reports... 2022-03-17T20:10:06.5867819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200951.xml 2022-03-17T20:10:09.2284801Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:10:09.2295684Z 2022-03-17T20:10:09.2296404Z Running tests... 2022-03-17T20:10:09.2297592Z ---------------------------------------------------------------------- 2022-03-17T20:10:09.8330962Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79694 2022-03-17T20:10:09.8365409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79695 2022-03-17T20:10:09.8366731Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:10:10.8924904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:10.9150169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:10.9151014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:10.9229413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:10.9229927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:11.0163388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:11.4482036Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:11.4804045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:13.1456244Z ok (3.915s) 2022-03-17T20:10:13.1456672Z 2022-03-17T20:10:13.1457907Z ---------------------------------------------------------------------- 2022-03-17T20:10:13.1458907Z Ran 1 test in 3.916s 2022-03-17T20:10:13.1459277Z 2022-03-17T20:10:13.1459467Z OK 2022-03-17T20:10:13.1460552Z 2022-03-17T20:10:13.1460972Z Generating XML reports... 2022-03-17T20:10:13.1539795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201009.xml 2022-03-17T20:10:15.9790255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:10:15.9800991Z 2022-03-17T20:10:15.9801310Z Running tests... 2022-03-17T20:10:15.9801742Z ---------------------------------------------------------------------- 2022-03-17T20:10:15.9820728Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:10:15.9821002Z 2022-03-17T20:10:15.9821303Z ---------------------------------------------------------------------- 2022-03-17T20:10:15.9821590Z Ran 1 test in 0.002s 2022-03-17T20:10:15.9821728Z 2022-03-17T20:10:15.9822115Z OK (skipped=1) 2022-03-17T20:10:15.9822291Z 2022-03-17T20:10:15.9822412Z Generating XML reports... 2022-03-17T20:10:15.9857353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201015.xml 2022-03-17T20:10:18.5059046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:10:18.5070944Z 2022-03-17T20:10:18.5071209Z Running tests... 2022-03-17T20:10:18.5071639Z ---------------------------------------------------------------------- 2022-03-17T20:10:18.5087927Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:10:19.1048290Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79976 2022-03-17T20:10:19.1085874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79977 2022-03-17T20:10:19.1087261Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:10:20.2320146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:20.2387833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:20.2388562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:20.2422997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:20.2425429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:20.3402483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:20.7218527Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:20.7741824Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:32.8420761Z ok (14.334s) 2022-03-17T20:10:32.8421422Z 2022-03-17T20:10:32.8422503Z ---------------------------------------------------------------------- 2022-03-17T20:10:32.8423286Z Ran 1 test in 14.335s 2022-03-17T20:10:32.8423655Z 2022-03-17T20:10:32.8423848Z OK 2022-03-17T20:10:32.8424159Z 2022-03-17T20:10:32.8424443Z Generating XML reports... 2022-03-17T20:10:32.8501947Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201018.xml 2022-03-17T20:10:35.4452143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:10:35.4463773Z 2022-03-17T20:10:35.4464037Z Running tests... 2022-03-17T20:10:35.4464585Z ---------------------------------------------------------------------- 2022-03-17T20:10:35.4486060Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:10:36.0626849Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80195 2022-03-17T20:10:36.0679483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80196 2022-03-17T20:10:36.0680787Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:10:37.2011002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:37.2051619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:37.2053534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:37.2114622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:37.2124374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:37.3061906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:37.6832435Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:37.7455790Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:47.3637732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:10:47.3644743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:10:49.9976916Z ok (14.551s) 2022-03-17T20:10:49.9977506Z 2022-03-17T20:10:49.9979018Z ---------------------------------------------------------------------- 2022-03-17T20:10:49.9980022Z Ran 1 test in 14.551s 2022-03-17T20:10:49.9980526Z 2022-03-17T20:10:49.9980749Z OK 2022-03-17T20:10:49.9981053Z 2022-03-17T20:10:49.9981353Z Generating XML reports... 2022-03-17T20:10:50.0062614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201035.xml 2022-03-17T20:10:52.6522258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:10:52.6534833Z 2022-03-17T20:10:52.6535205Z Running tests... 2022-03-17T20:10:52.6536159Z ---------------------------------------------------------------------- 2022-03-17T20:10:52.6556675Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:10:53.2525116Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80414 2022-03-17T20:10:53.2559335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80415 2022-03-17T20:10:53.2560621Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:10:54.3209562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:54.3804706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:54.3806694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:54.3819810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:54.3826394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:54.4816028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:54.8682794Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:10:54.9245473Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:04.6152314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:04.6153611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:07.2893259Z ok (14.635s) 2022-03-17T20:11:07.2893718Z 2022-03-17T20:11:07.2894801Z ---------------------------------------------------------------------- 2022-03-17T20:11:07.2895565Z Ran 1 test in 14.636s 2022-03-17T20:11:07.2895938Z 2022-03-17T20:11:07.2896124Z OK 2022-03-17T20:11:07.2896483Z 2022-03-17T20:11:07.2897302Z Generating XML reports... 2022-03-17T20:11:07.2976169Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201052.xml 2022-03-17T20:11:10.0183338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:10.0194853Z 2022-03-17T20:11:10.0195454Z Running tests... 2022-03-17T20:11:10.0197207Z ---------------------------------------------------------------------- 2022-03-17T20:11:10.0210821Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:10.6420349Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80633 2022-03-17T20:11:10.6473305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80634 2022-03-17T20:11:10.6474651Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:11:11.7163119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:11.7770987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:11.7773391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:11.7871266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:11.7872835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:11.8783853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:12.2608976Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:12.3066828Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:24.3752801Z ok (14.355s) 2022-03-17T20:11:24.3753152Z 2022-03-17T20:11:24.3754244Z ---------------------------------------------------------------------- 2022-03-17T20:11:24.3755143Z Ran 1 test in 14.356s 2022-03-17T20:11:24.3755550Z 2022-03-17T20:11:24.3755787Z OK 2022-03-17T20:11:24.3756117Z 2022-03-17T20:11:24.3756421Z Generating XML reports... 2022-03-17T20:11:24.3828588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201110.xml 2022-03-17T20:11:27.0396339Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:27.0409587Z 2022-03-17T20:11:27.0410168Z Running tests... 2022-03-17T20:11:27.0411344Z ---------------------------------------------------------------------- 2022-03-17T20:11:27.0423594Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:11:27.0424262Z 2022-03-17T20:11:27.0425402Z ---------------------------------------------------------------------- 2022-03-17T20:11:27.0426198Z Ran 1 test in 0.002s 2022-03-17T20:11:27.0426570Z 2022-03-17T20:11:27.0426805Z OK (skipped=1) 2022-03-17T20:11:27.0427153Z 2022-03-17T20:11:27.0427407Z Generating XML reports... 2022-03-17T20:11:27.0465990Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201127.xml 2022-03-17T20:11:29.7203013Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:29.7218407Z 2022-03-17T20:11:29.7218749Z Running tests... 2022-03-17T20:11:29.7219176Z ---------------------------------------------------------------------- 2022-03-17T20:11:29.7233581Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:11:29.7233855Z 2022-03-17T20:11:29.7234294Z ---------------------------------------------------------------------- 2022-03-17T20:11:29.7234606Z Ran 1 test in 0.002s 2022-03-17T20:11:29.7234756Z 2022-03-17T20:11:29.7234844Z OK (skipped=1) 2022-03-17T20:11:29.7234969Z 2022-03-17T20:11:29.7239605Z Generating XML reports... 2022-03-17T20:11:29.7272072Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201129.xml 2022-03-17T20:11:32.4457956Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:32.4469296Z 2022-03-17T20:11:32.4469594Z Running tests... 2022-03-17T20:11:32.4471850Z ---------------------------------------------------------------------- 2022-03-17T20:11:32.4486149Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:11:32.4487073Z 2022-03-17T20:11:32.4488233Z ---------------------------------------------------------------------- 2022-03-17T20:11:32.4489028Z Ran 1 test in 0.002s 2022-03-17T20:11:32.4489386Z 2022-03-17T20:11:32.4489626Z OK (skipped=1) 2022-03-17T20:11:32.4489971Z 2022-03-17T20:11:32.4490269Z Generating XML reports... 2022-03-17T20:11:32.4524610Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201132.xml 2022-03-17T20:11:35.0093469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:35.0104813Z 2022-03-17T20:11:35.0105174Z Running tests... 2022-03-17T20:11:35.0106587Z ---------------------------------------------------------------------- 2022-03-17T20:11:35.0123670Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:11:35.0124462Z 2022-03-17T20:11:35.0125473Z ---------------------------------------------------------------------- 2022-03-17T20:11:35.0126424Z Ran 1 test in 0.002s 2022-03-17T20:11:35.0126806Z 2022-03-17T20:11:35.0127039Z OK (skipped=1) 2022-03-17T20:11:35.0127388Z 2022-03-17T20:11:35.0127659Z Generating XML reports... 2022-03-17T20:11:35.0165488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201135.xml 2022-03-17T20:11:37.7468366Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:37.7478587Z 2022-03-17T20:11:37.7478694Z Running tests... 2022-03-17T20:11:37.7480363Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.7496438Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:11:37.7497172Z 2022-03-17T20:11:37.7497998Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.7498767Z Ran 1 test in 0.002s 2022-03-17T20:11:37.7499145Z 2022-03-17T20:11:37.7499382Z OK (skipped=1) 2022-03-17T20:11:37.7499740Z 2022-03-17T20:11:37.7500017Z Generating XML reports... 2022-03-17T20:11:37.7538851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201137.xml 2022-03-17T20:11:40.4698050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:40.4712547Z 2022-03-17T20:11:40.4712872Z Running tests... 2022-03-17T20:11:40.4714044Z ---------------------------------------------------------------------- 2022-03-17T20:11:40.4739114Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:11:40.4739747Z 2022-03-17T20:11:40.4740371Z ---------------------------------------------------------------------- 2022-03-17T20:11:40.4740837Z Ran 1 test in 0.003s 2022-03-17T20:11:40.4741046Z 2022-03-17T20:11:40.4741167Z OK (skipped=1) 2022-03-17T20:11:40.4741363Z 2022-03-17T20:11:40.4741529Z Generating XML reports... 2022-03-17T20:11:40.4777433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201140.xml 2022-03-17T20:11:43.0054568Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:43.0066813Z 2022-03-17T20:11:43.0067450Z Running tests... 2022-03-17T20:11:43.0068854Z ---------------------------------------------------------------------- 2022-03-17T20:11:43.0083613Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:11:43.0084578Z 2022-03-17T20:11:43.0085574Z ---------------------------------------------------------------------- 2022-03-17T20:11:43.0086522Z Ran 1 test in 0.002s 2022-03-17T20:11:43.0086988Z 2022-03-17T20:11:43.0087744Z OK (skipped=1) 2022-03-17T20:11:43.0088137Z 2022-03-17T20:11:43.0088398Z Generating XML reports... 2022-03-17T20:11:43.0120868Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201143.xml 2022-03-17T20:11:45.5675237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:45.5687312Z 2022-03-17T20:11:45.5687960Z Running tests... 2022-03-17T20:11:45.5689405Z ---------------------------------------------------------------------- 2022-03-17T20:11:46.1836355Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81321 2022-03-17T20:11:46.1879804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81322 2022-03-17T20:11:46.1881413Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:11:47.3235683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:47.3312473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:47.3314853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:47.3336316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:47.3338917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:47.4321440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:47.7119372Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:47.8104738Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:49.5977124Z ok (4.029s) 2022-03-17T20:11:49.5977633Z 2022-03-17T20:11:49.5978729Z ---------------------------------------------------------------------- 2022-03-17T20:11:49.5979716Z Ran 1 test in 4.029s 2022-03-17T20:11:49.5980178Z 2022-03-17T20:11:49.5980429Z OK 2022-03-17T20:11:49.5980800Z 2022-03-17T20:11:49.5981102Z Generating XML reports... 2022-03-17T20:11:49.6057444Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201145.xml 2022-03-17T20:11:52.3141875Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:52.3153481Z 2022-03-17T20:11:52.3153830Z Running tests... 2022-03-17T20:11:52.3154483Z ---------------------------------------------------------------------- 2022-03-17T20:11:52.9136268Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81542 2022-03-17T20:11:52.9172491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81543 2022-03-17T20:11:52.9174068Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:11:54.0446562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:54.0551917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:54.0554292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:54.0648900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:54.0651109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:54.1563575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:54.5379949Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:54.5862394Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:11:56.4284531Z ok (4.112s) 2022-03-17T20:11:56.4285141Z 2022-03-17T20:11:56.4286410Z ---------------------------------------------------------------------- 2022-03-17T20:11:56.4287428Z Ran 1 test in 4.113s 2022-03-17T20:11:56.4287918Z 2022-03-17T20:11:56.4288159Z OK 2022-03-17T20:11:56.4288514Z 2022-03-17T20:11:56.4288867Z Generating XML reports... 2022-03-17T20:11:56.4364793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201152.xml 2022-03-17T20:11:59.0829685Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:59.0842912Z 2022-03-17T20:11:59.0843624Z Running tests... 2022-03-17T20:11:59.0844400Z ---------------------------------------------------------------------- 2022-03-17T20:11:59.0859441Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:11:59.0859748Z 2022-03-17T20:11:59.0860122Z ---------------------------------------------------------------------- 2022-03-17T20:11:59.0860486Z Ran 1 test in 0.002s 2022-03-17T20:11:59.0860659Z 2022-03-17T20:11:59.0860764Z OK (skipped=1) 2022-03-17T20:11:59.0860913Z 2022-03-17T20:11:59.0861044Z Generating XML reports... 2022-03-17T20:11:59.0897576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201159.xml 2022-03-17T20:12:02.3525955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:02.3539055Z 2022-03-17T20:12:02.3539397Z Running tests... 2022-03-17T20:12:02.3539922Z ---------------------------------------------------------------------- 2022-03-17T20:12:02.3551330Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:12:02.3551629Z 2022-03-17T20:12:02.3551999Z ---------------------------------------------------------------------- 2022-03-17T20:12:02.3552363Z Ran 1 test in 0.002s 2022-03-17T20:12:02.3552541Z 2022-03-17T20:12:02.3552654Z OK (skipped=1) 2022-03-17T20:12:02.3552807Z 2022-03-17T20:12:02.3552932Z Generating XML reports... 2022-03-17T20:12:02.3590813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201202.xml 2022-03-17T20:12:04.8916184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:04.8927987Z 2022-03-17T20:12:04.8928207Z Running tests... 2022-03-17T20:12:04.8928760Z ---------------------------------------------------------------------- 2022-03-17T20:12:05.4924162Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81897 2022-03-17T20:12:05.4989758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81898 2022-03-17T20:12:05.4991423Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:12:06.6003081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:06.6264223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:06.6266535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:06.6310733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:06.6317675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:06.7274594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:07.0195654Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:07.1058683Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:09.1088454Z ok (4.215s) 2022-03-17T20:12:09.1089097Z 2022-03-17T20:12:09.1090180Z ---------------------------------------------------------------------- 2022-03-17T20:12:09.1091119Z Ran 1 test in 4.216s 2022-03-17T20:12:09.1091563Z 2022-03-17T20:12:09.1091818Z OK 2022-03-17T20:12:09.1092168Z 2022-03-17T20:12:09.1092443Z Generating XML reports... 2022-03-17T20:12:09.1169200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201204.xml 2022-03-17T20:12:11.6939049Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:11.6950446Z 2022-03-17T20:12:11.6950835Z Running tests... 2022-03-17T20:12:11.6952251Z ---------------------------------------------------------------------- 2022-03-17T20:12:12.3286167Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82118 2022-03-17T20:12:12.3330348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82119 2022-03-17T20:12:12.3331641Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:12:13.4085987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:13.4567923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:13.4569847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:13.4589716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:13.4592985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:13.5579949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:13.9450962Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:13.9989086Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:15.8436458Z ok (4.148s) 2022-03-17T20:12:15.8436996Z 2022-03-17T20:12:15.8438046Z ---------------------------------------------------------------------- 2022-03-17T20:12:15.8439025Z Ran 1 test in 4.148s 2022-03-17T20:12:15.8439482Z 2022-03-17T20:12:15.8439744Z OK 2022-03-17T20:12:15.8440053Z 2022-03-17T20:12:15.8440335Z Generating XML reports... 2022-03-17T20:12:15.8520486Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201211.xml 2022-03-17T20:12:18.4675483Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:18.4686885Z 2022-03-17T20:12:18.4687110Z Running tests... 2022-03-17T20:12:18.4687634Z ---------------------------------------------------------------------- 2022-03-17T20:12:19.1022707Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82339 2022-03-17T20:12:19.1069092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82340 2022-03-17T20:12:19.1070413Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:12:20.1889641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:20.2704277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:20.2706993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:20.2799746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:20.2807393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:20.3717719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:20.9963582Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:21.0515327Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:22.9176028Z ok (4.448s) 2022-03-17T20:12:22.9176340Z 2022-03-17T20:12:22.9177125Z ---------------------------------------------------------------------- 2022-03-17T20:12:22.9177729Z Ran 1 test in 4.449s 2022-03-17T20:12:22.9177998Z 2022-03-17T20:12:22.9178139Z OK 2022-03-17T20:12:22.9178338Z 2022-03-17T20:12:22.9178552Z Generating XML reports... 2022-03-17T20:12:22.9261474Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201218.xml 2022-03-17T20:12:25.5460265Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:25.5469449Z 2022-03-17T20:12:25.5469913Z Running tests... 2022-03-17T20:12:25.5471052Z ---------------------------------------------------------------------- 2022-03-17T20:12:26.1484874Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82554 2022-03-17T20:12:26.1519139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82555 2022-03-17T20:12:26.1520559Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:12:27.2189873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:27.2217131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:27.2218203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:27.2290354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:27.2291408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:27.3227809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:27.3440321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:12:27.3441018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:12:27.3442279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:12:27.3443090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:12:28.0567877Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:28.1065205Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:12:28.6936903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:12:28.6938902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:12:28.6940243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:12:28.6941947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:12:28.7340650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:12:28.7342572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:12:28.7343935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:12:28.7345653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:12:30.4623877Z ok (4.915s) 2022-03-17T20:12:30.4624275Z 2022-03-17T20:12:30.4625205Z ---------------------------------------------------------------------- 2022-03-17T20:12:30.4626018Z Ran 1 test in 4.915s 2022-03-17T20:12:30.4626392Z 2022-03-17T20:12:30.4626601Z OK 2022-03-17T20:12:30.4626874Z 2022-03-17T20:12:30.4627168Z Generating XML reports... 2022-03-17T20:12:30.4700363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201225.xml 2022-03-17T20:12:33.0663439Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:33.0675842Z 2022-03-17T20:12:33.0677115Z Running tests... 2022-03-17T20:12:33.0678559Z ---------------------------------------------------------------------- 2022-03-17T20:12:33.0711504Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:12:33.0712538Z 2022-03-17T20:12:33.0713627Z ---------------------------------------------------------------------- 2022-03-17T20:12:33.0715056Z Ran 1 test in 0.004s 2022-03-17T20:12:33.0715509Z 2022-03-17T20:12:33.0715811Z OK (skipped=1) 2022-03-17T20:12:33.0716236Z 2022-03-17T20:12:33.0716574Z Generating XML reports... 2022-03-17T20:12:33.0758669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201233.xml 2022-03-17T20:12:35.7928430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:35.7940619Z 2022-03-17T20:12:35.7940817Z Running tests... 2022-03-17T20:12:35.7941304Z ---------------------------------------------------------------------- 2022-03-17T20:12:35.7957558Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:35.7957940Z 2022-03-17T20:12:35.7958449Z ---------------------------------------------------------------------- 2022-03-17T20:12:35.7958792Z Ran 1 test in 0.002s 2022-03-17T20:12:35.7958977Z 2022-03-17T20:12:35.7959088Z OK (skipped=1) 2022-03-17T20:12:35.7959252Z 2022-03-17T20:12:35.7959373Z Generating XML reports... 2022-03-17T20:12:35.7995763Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201235.xml 2022-03-17T20:12:38.2804764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:38.2821148Z 2022-03-17T20:12:38.2821464Z Running tests... 2022-03-17T20:12:38.2821990Z ---------------------------------------------------------------------- 2022-03-17T20:12:38.2836860Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:38.2837193Z 2022-03-17T20:12:38.2837641Z ---------------------------------------------------------------------- 2022-03-17T20:12:38.2838113Z Ran 1 test in 0.002s 2022-03-17T20:12:38.2838341Z 2022-03-17T20:12:38.2838460Z OK (skipped=1) 2022-03-17T20:12:38.2838624Z 2022-03-17T20:12:38.2838761Z Generating XML reports... 2022-03-17T20:12:38.2875710Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201238.xml 2022-03-17T20:12:40.7576492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:40.7589584Z 2022-03-17T20:12:40.7589890Z Running tests... 2022-03-17T20:12:40.7590425Z ---------------------------------------------------------------------- 2022-03-17T20:12:40.7605089Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:40.7605430Z 2022-03-17T20:12:40.7605803Z ---------------------------------------------------------------------- 2022-03-17T20:12:40.7606150Z Ran 1 test in 0.002s 2022-03-17T20:12:40.7606322Z 2022-03-17T20:12:40.7606432Z OK (skipped=1) 2022-03-17T20:12:40.7606595Z 2022-03-17T20:12:40.7606714Z Generating XML reports... 2022-03-17T20:12:40.7644668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201240.xml 2022-03-17T20:12:43.3081123Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:43.3094299Z 2022-03-17T20:12:43.3094999Z Running tests... 2022-03-17T20:12:43.3096185Z ---------------------------------------------------------------------- 2022-03-17T20:12:43.3111593Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:43.3112322Z 2022-03-17T20:12:43.3113094Z ---------------------------------------------------------------------- 2022-03-17T20:12:43.3113875Z Ran 1 test in 0.002s 2022-03-17T20:12:43.3114240Z 2022-03-17T20:12:43.3114452Z OK (skipped=1) 2022-03-17T20:12:43.3114810Z 2022-03-17T20:12:43.3115085Z Generating XML reports... 2022-03-17T20:12:43.3149371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201243.xml 2022-03-17T20:12:45.7940470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:45.7950794Z 2022-03-17T20:12:45.7951051Z Running tests... 2022-03-17T20:12:45.7951588Z ---------------------------------------------------------------------- 2022-03-17T20:12:45.7969399Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:45.7969748Z 2022-03-17T20:12:45.7970105Z ---------------------------------------------------------------------- 2022-03-17T20:12:45.7970475Z Ran 1 test in 0.002s 2022-03-17T20:12:45.7970632Z 2022-03-17T20:12:45.7970741Z OK (skipped=1) 2022-03-17T20:12:45.7970914Z 2022-03-17T20:12:45.7971043Z Generating XML reports... 2022-03-17T20:12:45.8007305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201245.xml 2022-03-17T20:12:48.3112623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:48.3123113Z 2022-03-17T20:12:48.3123467Z Running tests... 2022-03-17T20:12:48.3123987Z ---------------------------------------------------------------------- 2022-03-17T20:12:48.3142007Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:48.3142874Z 2022-03-17T20:12:48.3143859Z ---------------------------------------------------------------------- 2022-03-17T20:12:48.3144783Z Ran 1 test in 0.002s 2022-03-17T20:12:48.3145158Z 2022-03-17T20:12:48.3145395Z OK (skipped=1) 2022-03-17T20:12:48.3145746Z 2022-03-17T20:12:48.3146018Z Generating XML reports... 2022-03-17T20:12:48.3182553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201248.xml 2022-03-17T20:12:50.8033190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:50.8043604Z 2022-03-17T20:12:50.8044387Z Running tests... 2022-03-17T20:12:50.8045583Z ---------------------------------------------------------------------- 2022-03-17T20:12:50.8060525Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:50.8061258Z 2022-03-17T20:12:50.8062067Z ---------------------------------------------------------------------- 2022-03-17T20:12:50.8062860Z Ran 1 test in 0.002s 2022-03-17T20:12:50.8063229Z 2022-03-17T20:12:50.8063453Z OK (skipped=1) 2022-03-17T20:12:50.8063804Z 2022-03-17T20:12:50.8064086Z Generating XML reports... 2022-03-17T20:12:50.8097007Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201250.xml 2022-03-17T20:12:53.3127058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:53.3137314Z 2022-03-17T20:12:53.3137761Z Running tests... 2022-03-17T20:12:53.3138892Z ---------------------------------------------------------------------- 2022-03-17T20:12:53.3156406Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:53.3157127Z 2022-03-17T20:12:53.3157935Z ---------------------------------------------------------------------- 2022-03-17T20:12:53.3158717Z Ran 1 test in 0.002s 2022-03-17T20:12:53.3159099Z 2022-03-17T20:12:53.3159344Z OK (skipped=1) 2022-03-17T20:12:53.3159698Z 2022-03-17T20:12:53.3159974Z Generating XML reports... 2022-03-17T20:12:53.3191632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201253.xml 2022-03-17T20:12:55.8386001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:55.8395294Z 2022-03-17T20:12:55.8395575Z Running tests... 2022-03-17T20:12:55.8396685Z ---------------------------------------------------------------------- 2022-03-17T20:12:55.8416110Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:55.8416873Z 2022-03-17T20:12:55.8418052Z ---------------------------------------------------------------------- 2022-03-17T20:12:55.8418826Z Ran 1 test in 0.002s 2022-03-17T20:12:55.8419197Z 2022-03-17T20:12:55.8419437Z OK (skipped=1) 2022-03-17T20:12:55.8419796Z 2022-03-17T20:12:55.8420056Z Generating XML reports... 2022-03-17T20:12:55.8459321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201255.xml 2022-03-17T20:12:58.5584180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:58.5595623Z 2022-03-17T20:12:58.5596265Z Running tests... 2022-03-17T20:12:58.5597428Z ---------------------------------------------------------------------- 2022-03-17T20:12:58.5616696Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:58.5617836Z 2022-03-17T20:12:58.5618835Z ---------------------------------------------------------------------- 2022-03-17T20:12:58.5619795Z Ran 1 test in 0.002s 2022-03-17T20:12:58.5620246Z 2022-03-17T20:12:58.5620536Z OK (skipped=1) 2022-03-17T20:12:58.5620961Z 2022-03-17T20:12:58.5621229Z Generating XML reports... 2022-03-17T20:12:58.5657480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201258.xml 2022-03-17T20:13:01.1095738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:01.1107988Z 2022-03-17T20:13:01.1108149Z Running tests... 2022-03-17T20:13:01.1108838Z ---------------------------------------------------------------------- 2022-03-17T20:13:01.1129072Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:01.1129536Z 2022-03-17T20:13:01.1130000Z ---------------------------------------------------------------------- 2022-03-17T20:13:01.1130426Z Ran 1 test in 0.002s 2022-03-17T20:13:01.1130639Z 2022-03-17T20:13:01.1130772Z OK (skipped=1) 2022-03-17T20:13:01.1130934Z 2022-03-17T20:13:01.1131068Z Generating XML reports... 2022-03-17T20:13:01.1171238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201301.xml 2022-03-17T20:13:03.7843998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:03.7855620Z 2022-03-17T20:13:03.7856241Z Running tests... 2022-03-17T20:13:03.7857407Z ---------------------------------------------------------------------- 2022-03-17T20:13:03.7873688Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:03.7874421Z 2022-03-17T20:13:03.7875228Z ---------------------------------------------------------------------- 2022-03-17T20:13:03.7876008Z Ran 1 test in 0.002s 2022-03-17T20:13:03.7876385Z 2022-03-17T20:13:03.7876616Z OK (skipped=1) 2022-03-17T20:13:03.7876986Z 2022-03-17T20:13:03.7877247Z Generating XML reports... 2022-03-17T20:13:03.7913782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201303.xml 2022-03-17T20:13:06.4869426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:06.4880538Z 2022-03-17T20:13:06.4881172Z Running tests... 2022-03-17T20:13:06.4882574Z ---------------------------------------------------------------------- 2022-03-17T20:13:06.4902708Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:06.4903617Z 2022-03-17T20:13:06.4904605Z ---------------------------------------------------------------------- 2022-03-17T20:13:06.4905574Z Ran 1 test in 0.002s 2022-03-17T20:13:06.4905930Z 2022-03-17T20:13:06.4906176Z OK (skipped=1) 2022-03-17T20:13:06.4906535Z 2022-03-17T20:13:06.4906813Z Generating XML reports... 2022-03-17T20:13:06.4942198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201306.xml 2022-03-17T20:13:09.0269717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:09.0281183Z 2022-03-17T20:13:09.0281436Z Running tests... 2022-03-17T20:13:09.0282046Z ---------------------------------------------------------------------- 2022-03-17T20:13:09.0298883Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:09.0299206Z 2022-03-17T20:13:09.0299570Z ---------------------------------------------------------------------- 2022-03-17T20:13:09.0299924Z Ran 1 test in 0.002s 2022-03-17T20:13:09.0300092Z 2022-03-17T20:13:09.0300203Z OK (skipped=1) 2022-03-17T20:13:09.0300367Z 2022-03-17T20:13:09.0300478Z Generating XML reports... 2022-03-17T20:13:09.0337374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201309.xml 2022-03-17T20:13:11.5213371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:11.5225597Z 2022-03-17T20:13:11.5226316Z Running tests... 2022-03-17T20:13:11.5227514Z ---------------------------------------------------------------------- 2022-03-17T20:13:11.5240606Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:11.5241283Z 2022-03-17T20:13:11.5242324Z ---------------------------------------------------------------------- 2022-03-17T20:13:11.5243150Z Ran 1 test in 0.002s 2022-03-17T20:13:11.5243507Z 2022-03-17T20:13:11.5243767Z OK (skipped=1) 2022-03-17T20:13:11.5244132Z 2022-03-17T20:13:11.5244406Z Generating XML reports... 2022-03-17T20:13:11.5281530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201311.xml 2022-03-17T20:13:13.9932348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:13.9943002Z 2022-03-17T20:13:13.9943467Z Running tests... 2022-03-17T20:13:13.9944574Z ---------------------------------------------------------------------- 2022-03-17T20:13:13.9962473Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:13.9963164Z 2022-03-17T20:13:13.9964739Z ---------------------------------------------------------------------- 2022-03-17T20:13:13.9965762Z Ran 1 test in 0.002s 2022-03-17T20:13:13.9966200Z 2022-03-17T20:13:13.9966497Z OK (skipped=1) 2022-03-17T20:13:13.9966973Z 2022-03-17T20:13:13.9967259Z Generating XML reports... 2022-03-17T20:13:14.0003574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201313.xml 2022-03-17T20:13:16.5288494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:16.5303004Z 2022-03-17T20:13:16.5303382Z Running tests... 2022-03-17T20:13:16.5304015Z ---------------------------------------------------------------------- 2022-03-17T20:13:16.5316812Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:16.5317231Z 2022-03-17T20:13:16.5317685Z ---------------------------------------------------------------------- 2022-03-17T20:13:16.5318112Z Ran 1 test in 0.002s 2022-03-17T20:13:16.5318325Z 2022-03-17T20:13:16.5318462Z OK (skipped=1) 2022-03-17T20:13:16.5318626Z 2022-03-17T20:13:16.5318745Z Generating XML reports... 2022-03-17T20:13:16.5358887Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201316.xml 2022-03-17T20:13:19.2782385Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:19.2795245Z 2022-03-17T20:13:19.2795918Z Running tests... 2022-03-17T20:13:19.2797415Z ---------------------------------------------------------------------- 2022-03-17T20:13:19.2815964Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:19.2816860Z 2022-03-17T20:13:19.2817816Z ---------------------------------------------------------------------- 2022-03-17T20:13:19.2819237Z Ran 1 test in 0.002s 2022-03-17T20:13:19.2819613Z 2022-03-17T20:13:19.2819857Z OK (skipped=1) 2022-03-17T20:13:19.2820199Z 2022-03-17T20:13:19.2820477Z Generating XML reports... 2022-03-17T20:13:19.2858522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201319.xml 2022-03-17T20:13:22.0652869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:22.0665150Z 2022-03-17T20:13:22.0665558Z Running tests... 2022-03-17T20:13:22.0666093Z ---------------------------------------------------------------------- 2022-03-17T20:13:22.0683322Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:22.0683651Z 2022-03-17T20:13:22.0684005Z ---------------------------------------------------------------------- 2022-03-17T20:13:22.0684365Z Ran 1 test in 0.002s 2022-03-17T20:13:22.0684535Z 2022-03-17T20:13:22.0684648Z OK (skipped=1) 2022-03-17T20:13:22.0684810Z 2022-03-17T20:13:22.0684948Z Generating XML reports... 2022-03-17T20:13:22.0721523Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201322.xml 2022-03-17T20:13:24.6549146Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:24.6560438Z 2022-03-17T20:13:24.6561464Z Running tests... 2022-03-17T20:13:24.6562837Z ---------------------------------------------------------------------- 2022-03-17T20:13:24.6580284Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:24.6580964Z 2022-03-17T20:13:24.6581744Z ---------------------------------------------------------------------- 2022-03-17T20:13:24.6582494Z Ran 1 test in 0.002s 2022-03-17T20:13:24.6582859Z 2022-03-17T20:13:24.6583093Z OK (skipped=1) 2022-03-17T20:13:24.6583450Z 2022-03-17T20:13:24.6583701Z Generating XML reports... 2022-03-17T20:13:24.6618734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201324.xml 2022-03-17T20:13:27.1761818Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:27.1775061Z 2022-03-17T20:13:27.1775222Z Running tests... 2022-03-17T20:13:27.1775767Z ---------------------------------------------------------------------- 2022-03-17T20:13:27.1792224Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:27.1792550Z 2022-03-17T20:13:27.1792909Z ---------------------------------------------------------------------- 2022-03-17T20:13:27.1793270Z Ran 1 test in 0.002s 2022-03-17T20:13:27.1793427Z 2022-03-17T20:13:27.1793536Z OK (skipped=1) 2022-03-17T20:13:27.1793700Z 2022-03-17T20:13:27.1793829Z Generating XML reports... 2022-03-17T20:13:27.1831161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201327.xml 2022-03-17T20:13:29.7322206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:29.7334488Z 2022-03-17T20:13:29.7335712Z Running tests... 2022-03-17T20:13:29.7336864Z ---------------------------------------------------------------------- 2022-03-17T20:13:29.7350449Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:29.7351085Z 2022-03-17T20:13:29.7351887Z ---------------------------------------------------------------------- 2022-03-17T20:13:29.7352637Z Ran 1 test in 0.002s 2022-03-17T20:13:29.7353000Z 2022-03-17T20:13:29.7353232Z OK (skipped=1) 2022-03-17T20:13:29.7353581Z 2022-03-17T20:13:29.7353858Z Generating XML reports... 2022-03-17T20:13:29.7390106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201329.xml 2022-03-17T20:13:32.3759441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:32.3769682Z 2022-03-17T20:13:32.3769996Z Running tests... 2022-03-17T20:13:32.3770655Z ---------------------------------------------------------------------- 2022-03-17T20:13:32.3787049Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:32.3787438Z 2022-03-17T20:13:32.3787946Z ---------------------------------------------------------------------- 2022-03-17T20:13:32.3788378Z Ran 1 test in 0.002s 2022-03-17T20:13:32.3788595Z 2022-03-17T20:13:32.3788744Z OK (skipped=1) 2022-03-17T20:13:32.3788940Z 2022-03-17T20:13:32.3789075Z Generating XML reports... 2022-03-17T20:13:32.3827553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201332.xml 2022-03-17T20:13:34.9758138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:34.9770309Z 2022-03-17T20:13:34.9770603Z Running tests... 2022-03-17T20:13:34.9771834Z ---------------------------------------------------------------------- 2022-03-17T20:13:34.9790337Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:34.9791005Z 2022-03-17T20:13:34.9791775Z ---------------------------------------------------------------------- 2022-03-17T20:13:34.9792553Z Ran 1 test in 0.002s 2022-03-17T20:13:34.9792925Z 2022-03-17T20:13:34.9793157Z OK (skipped=1) 2022-03-17T20:13:34.9793487Z 2022-03-17T20:13:34.9793774Z Generating XML reports... 2022-03-17T20:13:34.9828321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201334.xml 2022-03-17T20:13:37.4995905Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:37.5008804Z 2022-03-17T20:13:37.5009226Z Running tests... 2022-03-17T20:13:37.5010399Z ---------------------------------------------------------------------- 2022-03-17T20:13:37.5028427Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:37.5029122Z 2022-03-17T20:13:37.5029896Z ---------------------------------------------------------------------- 2022-03-17T20:13:37.5030666Z Ran 1 test in 0.002s 2022-03-17T20:13:37.5031031Z 2022-03-17T20:13:37.5031264Z OK (skipped=1) 2022-03-17T20:13:37.5031600Z 2022-03-17T20:13:37.5031871Z Generating XML reports... 2022-03-17T20:13:37.5068571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201337.xml 2022-03-17T20:13:39.9812992Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:39.9826201Z 2022-03-17T20:13:39.9826563Z Running tests... 2022-03-17T20:13:39.9827114Z ---------------------------------------------------------------------- 2022-03-17T20:13:39.9841743Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:39.9842145Z 2022-03-17T20:13:39.9842500Z ---------------------------------------------------------------------- 2022-03-17T20:13:39.9842858Z Ran 1 test in 0.002s 2022-03-17T20:13:39.9843034Z 2022-03-17T20:13:39.9843145Z OK (skipped=1) 2022-03-17T20:13:39.9843308Z 2022-03-17T20:13:39.9843436Z Generating XML reports... 2022-03-17T20:13:39.9882354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201339.xml 2022-03-17T20:13:42.4714276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:42.4725858Z 2022-03-17T20:13:42.4726405Z Running tests... 2022-03-17T20:13:42.4727596Z ---------------------------------------------------------------------- 2022-03-17T20:13:43.1016514Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84617 2022-03-17T20:13:43.1059859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84618 2022-03-17T20:13:43.1061841Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:13:44.2328346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:44.2400539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:44.2401640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:44.2431125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:44.2438423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:44.3411651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:44.7169224Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:13:44.7741395Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:13:46.8166384Z ok (4.344s) 2022-03-17T20:13:46.8167986Z 2022-03-17T20:13:46.8168890Z ---------------------------------------------------------------------- 2022-03-17T20:13:46.8169712Z Ran 1 test in 4.344s 2022-03-17T20:13:46.8170087Z 2022-03-17T20:13:46.8170261Z OK 2022-03-17T20:13:46.8170584Z 2022-03-17T20:13:46.8170886Z Generating XML reports... 2022-03-17T20:13:46.8257206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201342.xml 2022-03-17T20:13:49.6136455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:49.6150101Z 2022-03-17T20:13:49.6150630Z Running tests... 2022-03-17T20:13:49.6151782Z ---------------------------------------------------------------------- 2022-03-17T20:13:49.6164177Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:49.6164860Z 2022-03-17T20:13:49.6165639Z ---------------------------------------------------------------------- 2022-03-17T20:13:49.6166404Z Ran 1 test in 0.002s 2022-03-17T20:13:49.6166778Z 2022-03-17T20:13:49.6166992Z OK (skipped=1) 2022-03-17T20:13:49.6167341Z 2022-03-17T20:13:49.6167615Z Generating XML reports... 2022-03-17T20:13:49.6204782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201349.xml 2022-03-17T20:13:52.1807551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:52.1820651Z 2022-03-17T20:13:52.1821019Z Running tests... 2022-03-17T20:13:52.1821478Z ---------------------------------------------------------------------- 2022-03-17T20:13:52.1838523Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:52.1838774Z 2022-03-17T20:13:52.1839279Z ---------------------------------------------------------------------- 2022-03-17T20:13:52.1839583Z Ran 1 test in 0.002s 2022-03-17T20:13:52.1839727Z 2022-03-17T20:13:52.1839836Z OK (skipped=1) 2022-03-17T20:13:52.1839979Z 2022-03-17T20:13:52.1840073Z Generating XML reports... 2022-03-17T20:13:52.1879558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201352.xml 2022-03-17T20:13:54.7461771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:54.7476093Z 2022-03-17T20:13:54.7476624Z Running tests... 2022-03-17T20:13:54.7477774Z ---------------------------------------------------------------------- 2022-03-17T20:13:54.7497489Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:54.7498419Z 2022-03-17T20:13:54.7499533Z ---------------------------------------------------------------------- 2022-03-17T20:13:54.7500349Z Ran 1 test in 0.002s 2022-03-17T20:13:54.7501299Z 2022-03-17T20:13:54.7501588Z OK (skipped=1) 2022-03-17T20:13:54.7502012Z 2022-03-17T20:13:54.7502342Z Generating XML reports... 2022-03-17T20:13:54.7539394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201354.xml 2022-03-17T20:13:57.4828137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:57.4840163Z 2022-03-17T20:13:57.4840494Z Running tests... 2022-03-17T20:13:57.4841148Z ---------------------------------------------------------------------- 2022-03-17T20:13:58.0830890Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85033 2022-03-17T20:13:58.0862254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85034 2022-03-17T20:13:58.0863784Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:13:59.1501668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:59.2054281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:59.2056527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:59.2111131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:59.2112697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:59.3063728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:59.6841301Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:13:59.7302710Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:14:01.4954361Z ok (4.011s) 2022-03-17T20:14:01.4954910Z 2022-03-17T20:14:01.4956079Z ---------------------------------------------------------------------- 2022-03-17T20:14:01.4957100Z Ran 1 test in 4.011s 2022-03-17T20:14:01.4957550Z 2022-03-17T20:14:01.4957782Z OK 2022-03-17T20:14:01.4958117Z 2022-03-17T20:14:01.4958462Z Generating XML reports... 2022-03-17T20:14:01.5038090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201357.xml 2022-03-17T20:14:04.1149954Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:04.1165685Z 2022-03-17T20:14:04.1166284Z Running tests... 2022-03-17T20:14:04.1167719Z ---------------------------------------------------------------------- 2022-03-17T20:14:04.7240007Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85254 2022-03-17T20:14:04.7293013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85255 2022-03-17T20:14:04.7294580Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:14:05.8475158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:05.8689594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:05.8691735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:05.8777989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:05.8780629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:05.9704279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:06.3551602Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:14:06.4084330Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:14:08.5400401Z ok (4.423s) 2022-03-17T20:14:08.5400764Z 2022-03-17T20:14:08.5401706Z ---------------------------------------------------------------------- 2022-03-17T20:14:08.5402793Z Ran 1 test in 4.423s 2022-03-17T20:14:08.5403182Z 2022-03-17T20:14:08.5403344Z OK 2022-03-17T20:14:08.5403561Z 2022-03-17T20:14:08.5403775Z Generating XML reports... 2022-03-17T20:14:08.5468916Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201404.xml 2022-03-17T20:14:11.1862594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:11.1874463Z 2022-03-17T20:14:11.1874756Z Running tests... 2022-03-17T20:14:11.1875286Z ---------------------------------------------------------------------- 2022-03-17T20:14:11.7867750Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85475 2022-03-17T20:14:11.7904614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85476 2022-03-17T20:14:11.7905991Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:14:12.8729892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:12.9077809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:12.9078880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:12.9134668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:12.9137763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:13.0087798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:13.3833741Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:14:13.4346569Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:14:15.1007805Z ok (3.913s) 2022-03-17T20:14:15.1009010Z 2022-03-17T20:14:15.1009970Z ---------------------------------------------------------------------- 2022-03-17T20:14:15.1010824Z Ran 1 test in 3.913s 2022-03-17T20:14:15.1011206Z 2022-03-17T20:14:15.1011407Z OK 2022-03-17T20:14:15.1011697Z 2022-03-17T20:14:15.1011999Z Generating XML reports... 2022-03-17T20:14:15.1089449Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201411.xml 2022-03-17T20:14:17.8895824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:17.8908511Z 2022-03-17T20:14:17.8908669Z Running tests... 2022-03-17T20:14:17.8909201Z ---------------------------------------------------------------------- 2022-03-17T20:14:17.8923120Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:14:17.8923487Z 2022-03-17T20:14:17.8923857Z ---------------------------------------------------------------------- 2022-03-17T20:14:17.8924217Z Ran 1 test in 0.002s 2022-03-17T20:14:17.8924392Z 2022-03-17T20:14:17.8924503Z OK (skipped=1) 2022-03-17T20:14:17.8924668Z 2022-03-17T20:14:17.8924798Z Generating XML reports... 2022-03-17T20:14:17.8963273Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201417.xml 2022-03-17T20:14:20.4129415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:20.4141495Z 2022-03-17T20:14:20.4142379Z Running tests... 2022-03-17T20:14:20.4143562Z ---------------------------------------------------------------------- 2022-03-17T20:14:20.4158498Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:14:20.4159217Z 2022-03-17T20:14:20.4160014Z ---------------------------------------------------------------------- 2022-03-17T20:14:20.4160822Z Ran 1 test in 0.002s 2022-03-17T20:14:20.4161175Z 2022-03-17T20:14:20.4161415Z OK (skipped=1) 2022-03-17T20:14:20.4161783Z 2022-03-17T20:14:20.4162243Z Generating XML reports... 2022-03-17T20:14:20.4197895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201420.xml 2022-03-17T20:14:22.9566882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:22.9579240Z 2022-03-17T20:14:22.9579541Z Running tests... 2022-03-17T20:14:22.9580070Z ---------------------------------------------------------------------- 2022-03-17T20:14:23.5562918Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85830 2022-03-17T20:14:23.5599992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85831 2022-03-17T20:14:23.5601301Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:14:24.6204730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:24.6287923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:24.6290060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:24.6305954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:24.6309333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:24.7299095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:26.6681003Z skip (3.710s) 2022-03-17T20:14:26.6681517Z 2022-03-17T20:14:26.6682960Z ---------------------------------------------------------------------- 2022-03-17T20:14:26.6683946Z Ran 1 test in 3.710s 2022-03-17T20:14:26.6684403Z 2022-03-17T20:14:26.6685426Z OK (skipped=1) 2022-03-17T20:14:26.6685882Z 2022-03-17T20:14:26.6686218Z Generating XML reports... 2022-03-17T20:14:26.6765087Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201422.xml 2022-03-17T20:14:29.3662254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:29.3674577Z 2022-03-17T20:14:29.3675207Z Running tests... 2022-03-17T20:14:29.3676382Z ---------------------------------------------------------------------- 2022-03-17T20:14:29.9914017Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86038 2022-03-17T20:14:29.9971160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86039 2022-03-17T20:14:29.9972477Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:14:31.0900909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:31.0951003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:31.0952916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:31.1002387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:31.1004732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:31.1962363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:33.3071209Z skip (3.939s) 2022-03-17T20:14:33.3071728Z 2022-03-17T20:14:33.3073166Z ---------------------------------------------------------------------- 2022-03-17T20:14:33.3074509Z Ran 1 test in 3.939s 2022-03-17T20:14:33.3075165Z 2022-03-17T20:14:33.3075507Z OK (skipped=1) 2022-03-17T20:14:33.3076168Z 2022-03-17T20:14:33.3076625Z Generating XML reports... 2022-03-17T20:14:33.3150353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201429.xml 2022-03-17T20:14:36.0628391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:36.0640900Z 2022-03-17T20:14:36.0641028Z Running tests... 2022-03-17T20:14:36.0642091Z ---------------------------------------------------------------------- 2022-03-17T20:14:36.0658236Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:36.0658629Z 2022-03-17T20:14:36.0659106Z ---------------------------------------------------------------------- 2022-03-17T20:14:36.0659526Z Ran 1 test in 0.002s 2022-03-17T20:14:36.0659738Z 2022-03-17T20:14:36.0659870Z OK (skipped=1) 2022-03-17T20:14:36.0660081Z 2022-03-17T20:14:36.0660253Z Generating XML reports... 2022-03-17T20:14:36.0702017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201436.xml 2022-03-17T20:14:38.6591662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:38.6605540Z 2022-03-17T20:14:38.6605691Z Running tests... 2022-03-17T20:14:38.6606444Z ---------------------------------------------------------------------- 2022-03-17T20:14:39.2880895Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86313 2022-03-17T20:14:39.2932284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86314 2022-03-17T20:14:39.2933012Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:14:40.4358091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:40.4432521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:40.4434950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:40.4456385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:40.4458805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:40.5445538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:42.2031078Z skip (3.542s) 2022-03-17T20:14:42.2031506Z 2022-03-17T20:14:42.2032587Z ---------------------------------------------------------------------- 2022-03-17T20:14:42.2033358Z Ran 1 test in 3.542s 2022-03-17T20:14:42.2033727Z 2022-03-17T20:14:42.2033952Z OK (skipped=1) 2022-03-17T20:14:42.2034316Z 2022-03-17T20:14:42.2034606Z Generating XML reports... 2022-03-17T20:14:42.2108771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201438.xml 2022-03-17T20:14:44.8795348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:44.8807537Z 2022-03-17T20:14:44.8808518Z Running tests... 2022-03-17T20:14:44.8809453Z ---------------------------------------------------------------------- 2022-03-17T20:14:44.8822633Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:14:44.8822940Z 2022-03-17T20:14:44.8823376Z ---------------------------------------------------------------------- 2022-03-17T20:14:44.8823798Z Ran 1 test in 0.002s 2022-03-17T20:14:44.8824000Z 2022-03-17T20:14:44.8824105Z OK (skipped=1) 2022-03-17T20:14:44.8824287Z 2022-03-17T20:14:44.8824433Z Generating XML reports... 2022-03-17T20:14:44.8859907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201444.xml 2022-03-17T20:14:47.4078330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:47.4091495Z 2022-03-17T20:14:47.4091902Z Running tests... 2022-03-17T20:14:47.4093279Z ---------------------------------------------------------------------- 2022-03-17T20:14:48.0256461Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86588 2022-03-17T20:14:48.0300225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86589 2022-03-17T20:14:48.0301624Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:14:49.1725415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:49.1761069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:49.1763202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:49.1829691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:49.1836538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:49.2770262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:51.6395018Z skip (4.230s) 2022-03-17T20:14:51.6395472Z 2022-03-17T20:14:51.6396446Z ---------------------------------------------------------------------- 2022-03-17T20:14:51.6397333Z Ran 1 test in 4.230s 2022-03-17T20:14:51.6397769Z 2022-03-17T20:14:51.6398061Z OK (skipped=1) 2022-03-17T20:14:51.6398487Z 2022-03-17T20:14:51.6398805Z Generating XML reports... 2022-03-17T20:14:51.6472556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201447.xml 2022-03-17T20:14:54.3161272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:54.3173871Z 2022-03-17T20:14:54.3174232Z Running tests... 2022-03-17T20:14:54.3174771Z ---------------------------------------------------------------------- 2022-03-17T20:14:54.3191926Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:14:54.3192296Z 2022-03-17T20:14:54.3192767Z ---------------------------------------------------------------------- 2022-03-17T20:14:54.3193128Z Ran 1 test in 0.002s 2022-03-17T20:14:54.3193296Z 2022-03-17T20:14:54.3193397Z OK (skipped=1) 2022-03-17T20:14:54.3193552Z 2022-03-17T20:14:54.3193677Z Generating XML reports... 2022-03-17T20:14:54.3232742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201454.xml 2022-03-17T20:14:57.2351394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:57.2364579Z 2022-03-17T20:14:57.2364934Z Running tests... 2022-03-17T20:14:57.2365466Z ---------------------------------------------------------------------- 2022-03-17T20:14:57.2378580Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:14:57.2379045Z 2022-03-17T20:14:57.2379488Z ---------------------------------------------------------------------- 2022-03-17T20:14:57.2380190Z Ran 1 test in 0.002s 2022-03-17T20:14:57.2380362Z 2022-03-17T20:14:57.2380471Z OK (skipped=1) 2022-03-17T20:14:57.2380634Z 2022-03-17T20:14:57.2380752Z Generating XML reports... 2022-03-17T20:14:57.2421379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201457.xml 2022-03-17T20:14:59.6950686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:59.6964485Z 2022-03-17T20:14:59.6964774Z Running tests... 2022-03-17T20:14:59.6965548Z ---------------------------------------------------------------------- 2022-03-17T20:15:00.2992480Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86930 2022-03-17T20:15:00.3027939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86931 2022-03-17T20:15:00.3029523Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:01.3959535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:01.4294834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:01.4296768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:01.4362288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:01.4365232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:01.5306636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:01.8185811Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:01.9203380Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:04.1134550Z ok (4.416s) 2022-03-17T20:15:04.1134967Z 2022-03-17T20:15:04.1136042Z ---------------------------------------------------------------------- 2022-03-17T20:15:04.1136819Z Ran 1 test in 4.417s 2022-03-17T20:15:04.1137190Z 2022-03-17T20:15:04.1137374Z OK 2022-03-17T20:15:04.1137668Z 2022-03-17T20:15:04.1137937Z Generating XML reports... 2022-03-17T20:15:04.1215715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201459.xml 2022-03-17T20:15:06.9443246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:06.9456177Z 2022-03-17T20:15:06.9456427Z Running tests... 2022-03-17T20:15:06.9456986Z ---------------------------------------------------------------------- 2022-03-17T20:15:07.5820610Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87151 2022-03-17T20:15:07.5860156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87152 2022-03-17T20:15:07.5861575Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:08.6524488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:08.7074573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:08.7075835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:08.7132870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:08.7134193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:08.8085592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:09.1877956Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:09.2430189Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:11.0968042Z ok (4.150s) 2022-03-17T20:15:11.0968612Z 2022-03-17T20:15:11.0969634Z ---------------------------------------------------------------------- 2022-03-17T20:15:11.0970526Z Ran 1 test in 4.151s 2022-03-17T20:15:11.0970938Z 2022-03-17T20:15:11.0971125Z OK 2022-03-17T20:15:11.0971448Z 2022-03-17T20:15:11.0971754Z Generating XML reports... 2022-03-17T20:15:11.1053197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201506.xml 2022-03-17T20:15:13.7618432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:13.7627020Z 2022-03-17T20:15:13.7627321Z Running tests... 2022-03-17T20:15:13.7628541Z ---------------------------------------------------------------------- 2022-03-17T20:15:13.7643954Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:15:13.7644755Z 2022-03-17T20:15:13.7645649Z ---------------------------------------------------------------------- 2022-03-17T20:15:13.7646513Z Ran 1 test in 0.002s 2022-03-17T20:15:13.7646909Z 2022-03-17T20:15:13.7647189Z OK (skipped=1) 2022-03-17T20:15:13.7647809Z 2022-03-17T20:15:13.7648123Z Generating XML reports... 2022-03-17T20:15:13.7683929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201513.xml 2022-03-17T20:15:16.2342019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:16.2355413Z 2022-03-17T20:15:16.2355772Z Running tests... 2022-03-17T20:15:16.2356321Z ---------------------------------------------------------------------- 2022-03-17T20:15:16.8659781Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87439 2022-03-17T20:15:16.8726852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87440 2022-03-17T20:15:16.8728827Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:17.9970531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:18.0086767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:18.0088962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:18.0173847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:18.0182996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:18.0184190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:15:18.1106193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:18.1108809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:15:18.1110781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:15:18.1208508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:15:18.5025993Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:18.5642995Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:20.3817892Z ok (4.146s) 2022-03-17T20:15:20.3818528Z 2022-03-17T20:15:20.3819592Z ---------------------------------------------------------------------- 2022-03-17T20:15:20.3820387Z Ran 1 test in 4.146s 2022-03-17T20:15:20.3820756Z 2022-03-17T20:15:20.3820955Z OK 2022-03-17T20:15:20.3821253Z 2022-03-17T20:15:20.3821537Z Generating XML reports... 2022-03-17T20:15:20.3899324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201516.xml 2022-03-17T20:15:23.0356068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:23.0370234Z 2022-03-17T20:15:23.0370494Z Running tests... 2022-03-17T20:15:23.0370997Z ---------------------------------------------------------------------- 2022-03-17T20:15:23.0384633Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:15:23.0385330Z 2022-03-17T20:15:23.0386100Z ---------------------------------------------------------------------- 2022-03-17T20:15:23.0386824Z Ran 1 test in 0.002s 2022-03-17T20:15:23.0387101Z 2022-03-17T20:15:23.0387287Z OK (skipped=1) 2022-03-17T20:15:23.0387559Z 2022-03-17T20:15:23.0387777Z Generating XML reports... 2022-03-17T20:15:23.0425124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201523.xml 2022-03-17T20:15:25.8147484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:25.8155032Z 2022-03-17T20:15:25.8155295Z Running tests... 2022-03-17T20:15:25.8156369Z ---------------------------------------------------------------------- 2022-03-17T20:15:26.4122908Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87731 2022-03-17T20:15:26.4178998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87732 2022-03-17T20:15:26.4180480Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:27.4953095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:27.5065552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:27.5066657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:27.5156578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:27.5159385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:27.6075097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:29.5282255Z skip (3.712s) 2022-03-17T20:15:29.5282655Z 2022-03-17T20:15:29.5283488Z ---------------------------------------------------------------------- 2022-03-17T20:15:29.5284789Z Ran 1 test in 3.712s 2022-03-17T20:15:29.5285146Z 2022-03-17T20:15:29.5285382Z OK (skipped=1) 2022-03-17T20:15:29.5285729Z 2022-03-17T20:15:29.5286000Z Generating XML reports... 2022-03-17T20:15:29.5365048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201525.xml 2022-03-17T20:15:32.1196760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:32.1208021Z 2022-03-17T20:15:32.1208348Z Running tests... 2022-03-17T20:15:32.1208876Z ---------------------------------------------------------------------- 2022-03-17T20:15:32.1223510Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:15:32.1223775Z 2022-03-17T20:15:32.1224239Z ---------------------------------------------------------------------- 2022-03-17T20:15:32.1224890Z Ran 1 test in 0.002s 2022-03-17T20:15:32.1225033Z 2022-03-17T20:15:32.1225126Z OK (skipped=1) 2022-03-17T20:15:32.1225271Z 2022-03-17T20:15:32.1225394Z Generating XML reports... 2022-03-17T20:15:32.1262778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201532.xml 2022-03-17T20:15:34.6068868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:34.6087373Z 2022-03-17T20:15:34.6087688Z Running tests... 2022-03-17T20:15:34.6088331Z ---------------------------------------------------------------------- 2022-03-17T20:15:34.6102888Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:34.6103780Z 2022-03-17T20:15:34.6104761Z ---------------------------------------------------------------------- 2022-03-17T20:15:34.6105680Z Ran 1 test in 0.002s 2022-03-17T20:15:34.6106078Z 2022-03-17T20:15:34.6106301Z OK (skipped=1) 2022-03-17T20:15:34.6106653Z 2022-03-17T20:15:34.6106927Z Generating XML reports... 2022-03-17T20:15:34.6146335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201534.xml 2022-03-17T20:15:37.6217114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:37.6231852Z 2022-03-17T20:15:37.6232551Z Running tests... 2022-03-17T20:15:37.6233712Z ---------------------------------------------------------------------- 2022-03-17T20:15:38.2303137Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88073 2022-03-17T20:15:38.2338013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88074 2022-03-17T20:15:39.3133121Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:39.3133767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:39.3726065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:39.3726975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:39.3741907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:39.3746051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:39.4733883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:39.8504347Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:39.9037411Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:41.9435746Z ok (4.320s) 2022-03-17T20:15:41.9436220Z 2022-03-17T20:15:41.9437288Z ---------------------------------------------------------------------- 2022-03-17T20:15:41.9438079Z Ran 1 test in 4.320s 2022-03-17T20:15:41.9438450Z 2022-03-17T20:15:41.9438625Z OK 2022-03-17T20:15:41.9438921Z 2022-03-17T20:15:41.9439202Z Generating XML reports... 2022-03-17T20:15:41.9518955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201537.xml 2022-03-17T20:15:44.5255706Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:44.5266329Z 2022-03-17T20:15:44.5266529Z Running tests... 2022-03-17T20:15:44.5267067Z ---------------------------------------------------------------------- 2022-03-17T20:15:45.1662388Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88288 2022-03-17T20:15:45.1716421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88289 2022-03-17T20:15:45.1717984Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:46.2592585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:46.2875772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:46.2876710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:46.2896903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:46.2905070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:46.3883294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:46.7646719Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:46.8244072Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:48.7814950Z ok (4.254s) 2022-03-17T20:15:48.7815483Z 2022-03-17T20:15:48.7816730Z ---------------------------------------------------------------------- 2022-03-17T20:15:48.7817550Z Ran 1 test in 4.254s 2022-03-17T20:15:48.7817963Z 2022-03-17T20:15:48.7818163Z OK 2022-03-17T20:15:48.7818438Z 2022-03-17T20:15:48.7818721Z Generating XML reports... 2022-03-17T20:15:48.7895393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201544.xml 2022-03-17T20:15:51.3973733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:51.3983577Z 2022-03-17T20:15:51.3983736Z Running tests... 2022-03-17T20:15:51.3984290Z ---------------------------------------------------------------------- 2022-03-17T20:15:51.3999760Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:15:51.4000076Z 2022-03-17T20:15:51.4000443Z ---------------------------------------------------------------------- 2022-03-17T20:15:51.4000797Z Ran 1 test in 0.002s 2022-03-17T20:15:51.4000950Z 2022-03-17T20:15:51.4001058Z OK (skipped=1) 2022-03-17T20:15:51.4001220Z 2022-03-17T20:15:51.4001758Z Generating XML reports... 2022-03-17T20:15:51.4039453Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201551.xml 2022-03-17T20:15:53.9207756Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:53.9220515Z 2022-03-17T20:15:53.9220694Z Running tests... 2022-03-17T20:15:53.9221222Z ---------------------------------------------------------------------- 2022-03-17T20:15:54.5333602Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88570 2022-03-17T20:15:54.5384088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88571 2022-03-17T20:15:54.5385400Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:15:55.6014498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:55.6203331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:55.6204224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:55.6215490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:55.6219845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:55.6222103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:15:55.7216063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:55.7216890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:15:55.7217727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:15:55.7239359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:15:56.1034139Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:56.1534800Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:15:57.9480707Z ok (4.026s) 2022-03-17T20:15:57.9481317Z 2022-03-17T20:15:57.9482632Z ---------------------------------------------------------------------- 2022-03-17T20:15:57.9483526Z Ran 1 test in 4.026s 2022-03-17T20:15:57.9483905Z 2022-03-17T20:15:57.9484089Z OK 2022-03-17T20:15:57.9484372Z 2022-03-17T20:15:57.9484657Z Generating XML reports... 2022-03-17T20:15:57.9554472Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201553.xml 2022-03-17T20:16:00.6628082Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:00.6638195Z 2022-03-17T20:16:00.6638663Z Running tests... 2022-03-17T20:16:00.6639998Z ---------------------------------------------------------------------- 2022-03-17T20:16:00.6654536Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:16:00.6655697Z 2022-03-17T20:16:00.6656529Z ---------------------------------------------------------------------- 2022-03-17T20:16:00.6657322Z Ran 1 test in 0.002s 2022-03-17T20:16:00.6657673Z 2022-03-17T20:16:00.6657912Z OK (skipped=1) 2022-03-17T20:16:00.6658262Z 2022-03-17T20:16:00.6658534Z Generating XML reports... 2022-03-17T20:16:00.6692963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201600.xml 2022-03-17T20:16:03.1796242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:03.1811500Z 2022-03-17T20:16:03.1811696Z Running tests... 2022-03-17T20:16:03.1812437Z ---------------------------------------------------------------------- 2022-03-17T20:16:03.7784702Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88856 2022-03-17T20:16:03.7823744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88857 2022-03-17T20:16:03.7825125Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:16:04.9466407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:04.9826376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:04.9827537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:04.9869311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:04.9872984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:05.0837573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:06.6915394Z skip (3.510s) 2022-03-17T20:16:06.6915812Z 2022-03-17T20:16:06.6917240Z ---------------------------------------------------------------------- 2022-03-17T20:16:06.6918042Z Ran 1 test in 3.510s 2022-03-17T20:16:06.6918390Z 2022-03-17T20:16:06.6918618Z OK (skipped=1) 2022-03-17T20:16:06.6918959Z 2022-03-17T20:16:06.6919224Z Generating XML reports... 2022-03-17T20:16:06.6996327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201603.xml 2022-03-17T20:16:09.2284494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:09.2297426Z 2022-03-17T20:16:09.2298037Z Running tests... 2022-03-17T20:16:09.2299225Z ---------------------------------------------------------------------- 2022-03-17T20:16:09.8316497Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89064 2022-03-17T20:16:09.8349664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89065 2022-03-17T20:16:09.8350992Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:16:10.8943667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:10.9386975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:10.9387909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:10.9449174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:10.9452499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:11.0396707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:11.3761252Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:16:11.8216558Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:16:11.9880345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:16:11.9905906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:16:11.9907336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:16:11.9981246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:16:13.7439628Z ok (4.514s) 2022-03-17T20:16:13.7440295Z 2022-03-17T20:16:13.7441577Z ---------------------------------------------------------------------- 2022-03-17T20:16:13.7442746Z Ran 1 test in 4.514s 2022-03-17T20:16:13.7443106Z 2022-03-17T20:16:13.7443295Z OK 2022-03-17T20:16:13.7443642Z 2022-03-17T20:16:13.7443930Z Generating XML reports... 2022-03-17T20:16:13.7469689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201609.xml 2022-03-17T20:16:16.3801796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:16.3816077Z 2022-03-17T20:16:16.3816757Z Running tests... 2022-03-17T20:16:16.3818207Z ---------------------------------------------------------------------- 2022-03-17T20:16:16.9942883Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89290 2022-03-17T20:16:16.9988574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89291 2022-03-17T20:16:16.9990573Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:16:18.0887852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:18.1330663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:18.1332708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:18.1395646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:18.1398206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:18.2337378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:20.0079860Z skip (3.626s) 2022-03-17T20:16:20.0080475Z 2022-03-17T20:16:20.0081634Z ---------------------------------------------------------------------- 2022-03-17T20:16:20.0082815Z Ran 1 test in 3.626s 2022-03-17T20:16:20.0083265Z 2022-03-17T20:16:20.0083660Z OK (skipped=1) 2022-03-17T20:16:20.0084100Z 2022-03-17T20:16:20.0084356Z Generating XML reports... 2022-03-17T20:16:20.0168025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201616.xml 2022-03-17T20:16:22.9818033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:22.9827097Z 2022-03-17T20:16:22.9827191Z Running tests... 2022-03-17T20:16:22.9827794Z ---------------------------------------------------------------------- 2022-03-17T20:16:22.9845529Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:22.9845863Z 2022-03-17T20:16:22.9846367Z ---------------------------------------------------------------------- 2022-03-17T20:16:22.9846705Z Ran 1 test in 0.002s 2022-03-17T20:16:22.9846876Z 2022-03-17T20:16:22.9846979Z OK (skipped=1) 2022-03-17T20:16:22.9847141Z 2022-03-17T20:16:22.9847263Z Generating XML reports... 2022-03-17T20:16:22.9883115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201622.xml 2022-03-17T20:16:25.5499288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:25.5513240Z 2022-03-17T20:16:25.5513619Z Running tests... 2022-03-17T20:16:25.5514240Z ---------------------------------------------------------------------- 2022-03-17T20:16:25.5526909Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:16:25.5527613Z 2022-03-17T20:16:25.5528007Z ---------------------------------------------------------------------- 2022-03-17T20:16:25.5528369Z Ran 1 test in 0.002s 2022-03-17T20:16:25.5528548Z 2022-03-17T20:16:25.5528657Z OK (skipped=1) 2022-03-17T20:16:25.5528811Z 2022-03-17T20:16:25.5528937Z Generating XML reports... 2022-03-17T20:16:25.5565261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201625.xml 2022-03-17T20:16:28.0235569Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:28.0247931Z 2022-03-17T20:16:28.0248471Z Running tests... 2022-03-17T20:16:28.0249147Z ---------------------------------------------------------------------- 2022-03-17T20:16:28.6211542Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89632 2022-03-17T20:16:28.6247958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89633 2022-03-17T20:16:28.6249401Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:16:29.7128468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:29.7718568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:29.7720316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:29.7737204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:29.7744140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:29.8729853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:30.2539316Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:16:30.3118795Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:16:32.8395025Z ok (4.814s) 2022-03-17T20:16:32.8395395Z 2022-03-17T20:16:32.8396382Z ---------------------------------------------------------------------- 2022-03-17T20:16:32.8397148Z Ran 1 test in 4.814s 2022-03-17T20:16:32.8397506Z 2022-03-17T20:16:32.8397686Z OK 2022-03-17T20:16:32.8397960Z 2022-03-17T20:16:32.8398253Z Generating XML reports... 2022-03-17T20:16:32.8465278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201628.xml 2022-03-17T20:16:35.4924625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:35.4930982Z 2022-03-17T20:16:35.4931321Z Running tests... 2022-03-17T20:16:35.4931861Z ---------------------------------------------------------------------- 2022-03-17T20:16:35.4949434Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:35.4949954Z 2022-03-17T20:16:35.4950413Z ---------------------------------------------------------------------- 2022-03-17T20:16:35.4950926Z Ran 1 test in 0.002s 2022-03-17T20:16:35.4951125Z 2022-03-17T20:16:35.4951223Z OK (skipped=1) 2022-03-17T20:16:35.4951376Z 2022-03-17T20:16:35.4951503Z Generating XML reports... 2022-03-17T20:16:35.4986818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201635.xml 2022-03-17T20:16:37.9967412Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:37.9979912Z 2022-03-17T20:16:37.9980178Z Running tests... 2022-03-17T20:16:37.9980691Z ---------------------------------------------------------------------- 2022-03-17T20:16:38.5948994Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89914 2022-03-17T20:16:38.5981816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89915 2022-03-17T20:16:38.5983127Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:16:39.7195424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:39.7468658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:39.7469534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:39.7500000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:39.7505791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:39.8482691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:41.6075128Z skip (3.609s) 2022-03-17T20:16:41.6075657Z 2022-03-17T20:16:41.6076814Z ---------------------------------------------------------------------- 2022-03-17T20:16:41.6077781Z Ran 1 test in 3.609s 2022-03-17T20:16:41.6078229Z 2022-03-17T20:16:41.6078507Z OK (skipped=1) 2022-03-17T20:16:41.6078922Z 2022-03-17T20:16:41.6079259Z Generating XML reports... 2022-03-17T20:16:41.6150018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201637.xml 2022-03-17T20:16:44.2749719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:44.2763876Z 2022-03-17T20:16:44.2764603Z Running tests... 2022-03-17T20:16:44.2766097Z ---------------------------------------------------------------------- 2022-03-17T20:16:44.2777153Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:44.2777958Z 2022-03-17T20:16:44.2778879Z ---------------------------------------------------------------------- 2022-03-17T20:16:44.2779793Z Ran 1 test in 0.002s 2022-03-17T20:16:44.2780256Z 2022-03-17T20:16:44.2780499Z OK (skipped=1) 2022-03-17T20:16:44.2780850Z 2022-03-17T20:16:44.2781126Z Generating XML reports... 2022-03-17T20:16:44.2818766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201644.xml 2022-03-17T20:16:46.8542438Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:46.8558171Z 2022-03-17T20:16:46.8558705Z Running tests... 2022-03-17T20:16:46.8559904Z ---------------------------------------------------------------------- 2022-03-17T20:16:47.4545331Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90189 2022-03-17T20:16:47.4575400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90190 2022-03-17T20:16:47.4576804Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:16:48.5815598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:48.5830935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:48.5831864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:48.5916503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:48.5921650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:48.6846414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:50.4671337Z skip (3.611s) 2022-03-17T20:16:50.4671762Z 2022-03-17T20:16:50.4672702Z ---------------------------------------------------------------------- 2022-03-17T20:16:50.4673488Z Ran 1 test in 3.611s 2022-03-17T20:16:50.4673869Z 2022-03-17T20:16:50.4674121Z OK (skipped=1) 2022-03-17T20:16:50.4674484Z 2022-03-17T20:16:50.4674766Z Generating XML reports... 2022-03-17T20:16:50.4748218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201646.xml 2022-03-17T20:16:53.0579480Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:53.0591975Z 2022-03-17T20:16:53.0592211Z Running tests... 2022-03-17T20:16:53.0592636Z ---------------------------------------------------------------------- 2022-03-17T20:16:53.0610812Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:53.0611467Z 2022-03-17T20:16:53.0612276Z ---------------------------------------------------------------------- 2022-03-17T20:16:53.0613040Z Ran 1 test in 0.002s 2022-03-17T20:16:53.0613401Z 2022-03-17T20:16:53.0613613Z OK (skipped=1) 2022-03-17T20:16:53.0613954Z 2022-03-17T20:16:53.0614218Z Generating XML reports... 2022-03-17T20:16:53.0649195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201653.xml 2022-03-17T20:16:55.5820669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:55.5833074Z 2022-03-17T20:16:55.5833369Z Running tests... 2022-03-17T20:16:55.5833899Z ---------------------------------------------------------------------- 2022-03-17T20:16:55.5854999Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:55.5855384Z 2022-03-17T20:16:55.5855745Z ---------------------------------------------------------------------- 2022-03-17T20:16:55.5856187Z Ran 1 test in 0.002s 2022-03-17T20:16:55.5856382Z 2022-03-17T20:16:55.5856488Z OK (skipped=1) 2022-03-17T20:16:55.5856643Z 2022-03-17T20:16:55.5856764Z Generating XML reports... 2022-03-17T20:16:55.5894318Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201655.xml 2022-03-17T20:16:58.1531913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:58.1545288Z 2022-03-17T20:16:58.1545719Z Running tests... 2022-03-17T20:16:58.1546912Z ---------------------------------------------------------------------- 2022-03-17T20:16:58.1568017Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:58.1568934Z 2022-03-17T20:16:58.1569732Z ---------------------------------------------------------------------- 2022-03-17T20:16:58.1570495Z Ran 1 test in 0.002s 2022-03-17T20:16:58.1570862Z 2022-03-17T20:16:58.1571074Z OK (skipped=1) 2022-03-17T20:16:58.1571420Z 2022-03-17T20:16:58.1571701Z Generating XML reports... 2022-03-17T20:16:58.1612271Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201658.xml 2022-03-17T20:17:00.8832857Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:00.8844041Z 2022-03-17T20:17:00.8844580Z Running tests... 2022-03-17T20:17:00.8845779Z ---------------------------------------------------------------------- 2022-03-17T20:17:00.8867352Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:00.8868021Z 2022-03-17T20:17:00.8868820Z ---------------------------------------------------------------------- 2022-03-17T20:17:00.8869615Z Ran 1 test in 0.002s 2022-03-17T20:17:00.8869986Z 2022-03-17T20:17:00.8870206Z OK (skipped=1) 2022-03-17T20:17:00.8870553Z 2022-03-17T20:17:00.8870833Z Generating XML reports... 2022-03-17T20:17:00.8906342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201700.xml 2022-03-17T20:17:03.4966889Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:03.4982054Z 2022-03-17T20:17:03.4982668Z Running tests... 2022-03-17T20:17:03.4983858Z ---------------------------------------------------------------------- 2022-03-17T20:17:03.5007303Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:17:03.5008286Z 2022-03-17T20:17:03.5009133Z ---------------------------------------------------------------------- 2022-03-17T20:17:03.5009904Z Ran 1 test in 0.003s 2022-03-17T20:17:03.5010253Z 2022-03-17T20:17:03.5010482Z OK (skipped=1) 2022-03-17T20:17:03.5010822Z 2022-03-17T20:17:03.5011092Z Generating XML reports... 2022-03-17T20:17:03.5056730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201703.xml 2022-03-17T20:17:06.1481155Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:06.1493868Z 2022-03-17T20:17:06.1494325Z Running tests... 2022-03-17T20:17:06.1495458Z ---------------------------------------------------------------------- 2022-03-17T20:17:06.7479364Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90732 2022-03-17T20:17:06.7512984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90733 2022-03-17T20:17:06.7514726Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:07.8843545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:07.8970388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:07.8972416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:07.9048700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:07.9059736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:07.9989078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:08.0189185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:17:08.0191114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:08.0192425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:17:08.0194094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:08.0195741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:17:08.0299044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:17:08.0301162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:17:08.0302980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:17:09.8596916Z ok (3.710s) 2022-03-17T20:17:09.8597296Z 2022-03-17T20:17:09.8598212Z ---------------------------------------------------------------------- 2022-03-17T20:17:09.8599003Z Ran 1 test in 3.710s 2022-03-17T20:17:09.8599366Z 2022-03-17T20:17:09.8599534Z OK 2022-03-17T20:17:09.8599822Z 2022-03-17T20:17:09.8600099Z Generating XML reports... 2022-03-17T20:17:09.8678869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201706.xml 2022-03-17T20:17:12.4220345Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:12.4232022Z 2022-03-17T20:17:12.4232654Z Running tests... 2022-03-17T20:17:12.4233204Z ---------------------------------------------------------------------- 2022-03-17T20:17:13.0192875Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90950 2022-03-17T20:17:13.0227118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90951 2022-03-17T20:17:13.0228444Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:14.1433805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:14.1445581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:14.1447503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:14.1536640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:14.1537995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:14.2460166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:14.6297576Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:17:14.7441460Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:17:16.6332730Z ok (4.209s) 2022-03-17T20:17:16.6333126Z 2022-03-17T20:17:16.6334429Z ---------------------------------------------------------------------- 2022-03-17T20:17:16.6335279Z Ran 1 test in 4.210s 2022-03-17T20:17:16.6335667Z 2022-03-17T20:17:16.6335853Z OK 2022-03-17T20:17:16.6336146Z 2022-03-17T20:17:16.6336416Z Generating XML reports... 2022-03-17T20:17:16.6398012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201712.xml 2022-03-17T20:17:19.3078149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:19.3090004Z 2022-03-17T20:17:19.3090364Z Running tests... 2022-03-17T20:17:19.3090918Z ---------------------------------------------------------------------- 2022-03-17T20:17:19.9054395Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91173 2022-03-17T20:17:19.9098862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91174 2022-03-17T20:17:19.9100189Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:21.0039772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:21.0496368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:21.0498343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:21.0546944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:21.0552279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:21.1509144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:22.9192838Z skip (3.610s) 2022-03-17T20:17:22.9193227Z 2022-03-17T20:17:22.9194444Z ---------------------------------------------------------------------- 2022-03-17T20:17:22.9196051Z Ran 1 test in 3.610s 2022-03-17T20:17:22.9196439Z 2022-03-17T20:17:22.9196688Z OK (skipped=1) 2022-03-17T20:17:22.9197028Z 2022-03-17T20:17:22.9197298Z Generating XML reports... 2022-03-17T20:17:22.9274641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201719.xml 2022-03-17T20:17:25.5318362Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:25.5335187Z 2022-03-17T20:17:25.5335545Z Running tests... 2022-03-17T20:17:25.5336052Z ---------------------------------------------------------------------- 2022-03-17T20:17:26.1324001Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91381 2022-03-17T20:17:26.1359906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91382 2022-03-17T20:17:26.1361201Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:27.1990951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:27.2042681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:27.2043566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:27.2092908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:27.2093665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:27.3052950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:29.5462036Z ok (4.013s) 2022-03-17T20:17:29.5462457Z 2022-03-17T20:17:29.5463510Z ---------------------------------------------------------------------- 2022-03-17T20:17:29.5464541Z Ran 1 test in 4.013s 2022-03-17T20:17:29.5464978Z 2022-03-17T20:17:29.5465206Z OK 2022-03-17T20:17:29.5465537Z 2022-03-17T20:17:29.5465872Z Generating XML reports... 2022-03-17T20:17:29.5539454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201725.xml 2022-03-17T20:17:32.1085954Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:32.1097318Z 2022-03-17T20:17:32.1097523Z Running tests... 2022-03-17T20:17:32.7089461Z ---------------------------------------------------------------------- 2022-03-17T20:17:32.7091074Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91589 2022-03-17T20:17:32.7125273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91590 2022-03-17T20:17:32.7126882Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:33.8355920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:33.8509248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:33.8511163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:33.8555500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:33.8557210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:33.9519955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:35.7202482Z ok (3.610s) 2022-03-17T20:17:35.7202883Z 2022-03-17T20:17:35.7204107Z ---------------------------------------------------------------------- 2022-03-17T20:17:35.7204914Z Ran 1 test in 3.610s 2022-03-17T20:17:35.7205312Z 2022-03-17T20:17:35.7205495Z OK 2022-03-17T20:17:35.7205788Z 2022-03-17T20:17:35.7206065Z Generating XML reports... 2022-03-17T20:17:35.7284098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201732.xml 2022-03-17T20:17:38.3524790Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:38.3539844Z 2022-03-17T20:17:38.3540455Z Running tests... 2022-03-17T20:17:38.3541646Z ---------------------------------------------------------------------- 2022-03-17T20:17:38.9816920Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91797 2022-03-17T20:17:38.9858324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91798 2022-03-17T20:17:38.9859705Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:40.0461427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:40.0839609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:40.0842084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:40.0865955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:40.0869546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:40.1851804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:40.5667090Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:17:42.2949237Z ok (3.941s) 2022-03-17T20:17:42.2949725Z 2022-03-17T20:17:42.2951472Z ---------------------------------------------------------------------- 2022-03-17T20:17:42.2952319Z Ran 1 test in 3.941s 2022-03-17T20:17:42.2952700Z 2022-03-17T20:17:42.2952868Z OK 2022-03-17T20:17:42.2954343Z 2022-03-17T20:17:42.2955008Z Generating XML reports... 2022-03-17T20:17:42.3024870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201738.xml 2022-03-17T20:17:44.8689517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:44.8704764Z 2022-03-17T20:17:44.8705461Z Running tests... 2022-03-17T20:17:44.8706640Z ---------------------------------------------------------------------- 2022-03-17T20:17:45.4820304Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92009 2022-03-17T20:17:45.4851941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92010 2022-03-17T20:17:45.4853231Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:46.5922395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:46.5988132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:46.5988930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:46.6026457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:46.6028430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:46.6995663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:48.7955394Z ok (3.925s) 2022-03-17T20:17:48.7955794Z 2022-03-17T20:17:48.7956645Z ---------------------------------------------------------------------- 2022-03-17T20:17:48.7957414Z Ran 1 test in 3.925s 2022-03-17T20:17:48.7957762Z 2022-03-17T20:17:48.7957956Z OK 2022-03-17T20:17:48.7958243Z 2022-03-17T20:17:48.7958514Z Generating XML reports... 2022-03-17T20:17:48.8038178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201744.xml 2022-03-17T20:17:51.3480427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:51.3491778Z 2022-03-17T20:17:51.3491907Z Running tests... 2022-03-17T20:17:51.3492578Z ---------------------------------------------------------------------- 2022-03-17T20:17:51.3509556Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:17:51.3510044Z 2022-03-17T20:17:51.3510518Z ---------------------------------------------------------------------- 2022-03-17T20:17:51.3510950Z Ran 1 test in 0.002s 2022-03-17T20:17:51.3511153Z 2022-03-17T20:17:51.3511281Z OK (skipped=1) 2022-03-17T20:17:51.3511462Z 2022-03-17T20:17:51.3511581Z Generating XML reports... 2022-03-17T20:17:51.3544834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201751.xml 2022-03-17T20:17:53.9775456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:53.9787171Z 2022-03-17T20:17:53.9787279Z Running tests... 2022-03-17T20:17:53.9787793Z ---------------------------------------------------------------------- 2022-03-17T20:17:54.5765394Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92284 2022-03-17T20:17:54.5802502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92285 2022-03-17T20:17:54.5803907Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:17:55.6975330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:55.7033328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:55.7034069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:55.7078742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:55.7082206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:55.8047188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:56.1844461Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:17:56.2395336Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:17:58.0913091Z ok (4.112s) 2022-03-17T20:17:58.0913506Z 2022-03-17T20:17:58.0914725Z ---------------------------------------------------------------------- 2022-03-17T20:17:58.0915661Z Ran 1 test in 4.112s 2022-03-17T20:17:58.0916109Z 2022-03-17T20:17:58.0916344Z OK 2022-03-17T20:17:58.0916706Z 2022-03-17T20:17:58.0917139Z Generating XML reports... 2022-03-17T20:17:58.0984788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201753.xml 2022-03-17T20:18:00.7783179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:00.7795443Z 2022-03-17T20:18:00.7796104Z Running tests... 2022-03-17T20:18:00.7797266Z ---------------------------------------------------------------------- 2022-03-17T20:18:00.7814918Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:18:00.7815806Z 2022-03-17T20:18:00.7816791Z ---------------------------------------------------------------------- 2022-03-17T20:18:00.7817755Z Ran 1 test in 0.002s 2022-03-17T20:18:00.7818138Z 2022-03-17T20:18:00.7818365Z OK (skipped=1) 2022-03-17T20:18:00.7818712Z 2022-03-17T20:18:00.7818978Z Generating XML reports... 2022-03-17T20:18:00.7850459Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201800.xml 2022-03-17T20:18:03.3299463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:03.3311150Z 2022-03-17T20:18:03.3311785Z Running tests... 2022-03-17T20:18:03.3313008Z ---------------------------------------------------------------------- 2022-03-17T20:18:03.3326661Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:18:03.3327198Z 2022-03-17T20:18:03.3327781Z ---------------------------------------------------------------------- 2022-03-17T20:18:03.3328137Z Ran 1 test in 0.002s 2022-03-17T20:18:03.3328300Z 2022-03-17T20:18:03.3328412Z OK (skipped=1) 2022-03-17T20:18:03.3328587Z 2022-03-17T20:18:03.3328707Z Generating XML reports... 2022-03-17T20:18:03.3365697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201803.xml 2022-03-17T20:18:05.8897291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:05.8909036Z 2022-03-17T20:18:05.8909440Z Running tests... 2022-03-17T20:18:05.8910751Z ---------------------------------------------------------------------- 2022-03-17T20:18:05.8926415Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:18:05.8927187Z 2022-03-17T20:18:05.8928229Z ---------------------------------------------------------------------- 2022-03-17T20:18:05.8929203Z Ran 1 test in 0.002s 2022-03-17T20:18:05.8929646Z 2022-03-17T20:18:05.8929965Z OK (skipped=1) 2022-03-17T20:18:05.8930371Z 2022-03-17T20:18:05.8930654Z Generating XML reports... 2022-03-17T20:18:05.8967160Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201805.xml 2022-03-17T20:18:08.4350328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:08.4362061Z 2022-03-17T20:18:08.4362762Z Running tests... 2022-03-17T20:18:08.4364229Z ---------------------------------------------------------------------- 2022-03-17T20:18:09.0289024Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92706 2022-03-17T20:18:09.0320252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92707 2022-03-17T20:18:09.0324585Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:18:10.0957225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:10.0991619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:10.0993597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:10.1059838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:10.1061464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:10.2004968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:10.8305142Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:18:10.8915212Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:18:12.7458264Z ok (4.309s) 2022-03-17T20:18:12.7458683Z 2022-03-17T20:18:12.7459866Z ---------------------------------------------------------------------- 2022-03-17T20:18:12.7460644Z Ran 1 test in 4.310s 2022-03-17T20:18:12.7461007Z 2022-03-17T20:18:12.7461188Z OK 2022-03-17T20:18:12.7461461Z 2022-03-17T20:18:12.7461750Z Generating XML reports... 2022-03-17T20:18:12.7540184Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201808.xml 2022-03-17T20:18:15.4014004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:15.4029161Z 2022-03-17T20:18:15.4029752Z Running tests... 2022-03-17T20:18:15.4030948Z ---------------------------------------------------------------------- 2022-03-17T20:18:16.0363545Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92921 2022-03-17T20:18:16.0399918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92922 2022-03-17T20:18:16.0401269Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:18:17.1120331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:17.1348512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:17.1350534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:17.1422281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:17.1428596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:17.2362324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:18.9492627Z skip (3.546s) 2022-03-17T20:18:18.9493381Z 2022-03-17T20:18:18.9494686Z ---------------------------------------------------------------------- 2022-03-17T20:18:18.9495649Z Ran 1 test in 3.546s 2022-03-17T20:18:18.9496031Z 2022-03-17T20:18:18.9496272Z OK (skipped=1) 2022-03-17T20:18:18.9496645Z 2022-03-17T20:18:18.9496923Z Generating XML reports... 2022-03-17T20:18:18.9575955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201815.xml 2022-03-17T20:18:21.5588006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:21.5603880Z 2022-03-17T20:18:21.5604191Z Running tests... 2022-03-17T20:18:21.5604715Z ---------------------------------------------------------------------- 2022-03-17T20:18:22.1907554Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93129 2022-03-17T20:18:22.1938605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93130 2022-03-17T20:18:22.1939994Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:18:23.2589982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:23.3185881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:23.3187541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:23.3196621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:23.3203400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:23.4197448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:25.1017937Z skip (3.541s) 2022-03-17T20:18:25.1018639Z 2022-03-17T20:18:25.1019635Z ---------------------------------------------------------------------- 2022-03-17T20:18:25.1020447Z Ran 1 test in 3.541s 2022-03-17T20:18:25.1020833Z 2022-03-17T20:18:25.1021064Z OK (skipped=1) 2022-03-17T20:18:25.1021396Z 2022-03-17T20:18:25.1021670Z Generating XML reports... 2022-03-17T20:18:25.1099634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201821.xml 2022-03-17T20:18:27.7885208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:27.7897987Z 2022-03-17T20:18:27.7898181Z Running tests... 2022-03-17T20:18:27.7898614Z ---------------------------------------------------------------------- 2022-03-17T20:18:28.3889250Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93337 2022-03-17T20:18:28.3927178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93338 2022-03-17T20:18:28.3927913Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:18:29.4830369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:29.5267138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:29.5268035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:29.5341618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:29.5346621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:29.6273908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:29.9186245Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:18:30.0144155Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:18:39.2650493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:39.3450334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:42.0182899Z ok (14.228s) 2022-03-17T20:18:42.0183426Z 2022-03-17T20:18:42.0184599Z ---------------------------------------------------------------------- 2022-03-17T20:18:42.0185570Z Ran 1 test in 14.228s 2022-03-17T20:18:42.0186017Z 2022-03-17T20:18:42.0186213Z OK 2022-03-17T20:18:42.0186757Z 2022-03-17T20:18:42.0187114Z Generating XML reports... 2022-03-17T20:18:42.0263289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201827.xml 2022-03-17T20:18:44.7375593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:44.7389034Z 2022-03-17T20:18:44.7389526Z Running tests... 2022-03-17T20:18:44.7390707Z ---------------------------------------------------------------------- 2022-03-17T20:18:45.3642817Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93556 2022-03-17T20:18:45.3681107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93557 2022-03-17T20:18:45.3682611Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:18:46.5222822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:46.5224588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:46.5226655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:46.5323531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:46.5325444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:46.6241639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:46.9155357Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:18:47.0086851Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:18:56.3288180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:56.3289806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:56.3294451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:56.3296043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:58.9900384Z ok (14.251s) 2022-03-17T20:18:58.9900782Z 2022-03-17T20:18:58.9902010Z ---------------------------------------------------------------------- 2022-03-17T20:18:58.9902818Z Ran 1 test in 14.251s 2022-03-17T20:18:58.9903673Z 2022-03-17T20:18:58.9903889Z OK 2022-03-17T20:18:58.9904200Z 2022-03-17T20:18:58.9904489Z Generating XML reports... 2022-03-17T20:18:58.9979015Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201844.xml 2022-03-17T20:19:01.6246273Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:01.6261421Z 2022-03-17T20:19:01.6261681Z Running tests... 2022-03-17T20:19:01.6262120Z ---------------------------------------------------------------------- 2022-03-17T20:19:02.2488312Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93775 2022-03-17T20:19:02.2524809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93776 2022-03-17T20:19:02.2525499Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:19:03.3506144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:03.3910665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:03.3911689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:03.3912519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:03.3916428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:03.3921134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:03.7719408Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:03.8299882Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:13.0592880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0596451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0620692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0622177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0808911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0810510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0812140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:13.0813770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:15.6852970Z ok (14.059s) 2022-03-17T20:19:15.6853514Z 2022-03-17T20:19:15.6854517Z ---------------------------------------------------------------------- 2022-03-17T20:19:15.6855349Z Ran 1 test in 14.059s 2022-03-17T20:19:15.6855726Z 2022-03-17T20:19:15.6855900Z OK 2022-03-17T20:19:15.6856200Z 2022-03-17T20:19:15.6856862Z Generating XML reports... 2022-03-17T20:19:15.6937448Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201901.xml 2022-03-17T20:19:18.2844179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:18.2859466Z 2022-03-17T20:19:18.2859747Z Running tests... 2022-03-17T20:19:18.2860181Z ---------------------------------------------------------------------- 2022-03-17T20:19:18.9336191Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93994 2022-03-17T20:19:18.9377765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93995 2022-03-17T20:19:18.9379545Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:19:20.0507714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:20.0520944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:20.0521988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:20.0608656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:20.0610573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:20.1535922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:20.5358162Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:20.5918674Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:29.7735761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.7737446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.7758478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.7760114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.8591024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.8593637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.8595971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:29.8597926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:32.5697238Z ok (14.283s) 2022-03-17T20:19:32.5697743Z 2022-03-17T20:19:32.5698884Z ---------------------------------------------------------------------- 2022-03-17T20:19:32.5699810Z Ran 1 test in 14.284s 2022-03-17T20:19:32.5700243Z 2022-03-17T20:19:32.5700478Z OK 2022-03-17T20:19:32.5700848Z 2022-03-17T20:19:32.5701124Z Generating XML reports... 2022-03-17T20:19:32.5777674Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201918.xml 2022-03-17T20:19:35.2053458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:35.2064779Z 2022-03-17T20:19:35.2065211Z Running tests... 2022-03-17T20:19:35.2065728Z ---------------------------------------------------------------------- 2022-03-17T20:19:35.8075469Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94213 2022-03-17T20:19:35.8111981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94214 2022-03-17T20:19:35.8113326Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:19:36.8885952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:36.9339321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:36.9341347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:36.9391764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:36.9396062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:37.0351955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:37.4225464Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:37.4728942Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:37.5786488Z 2022-03-17T20:19:39.2189016Z ok (4.012s) 2022-03-17T20:19:39.2189553Z 2022-03-17T20:19:39.2190676Z ---------------------------------------------------------------------- 2022-03-17T20:19:39.2191584Z Ran 1 test in 4.012s 2022-03-17T20:19:39.2191969Z 2022-03-17T20:19:39.2192146Z OK 2022-03-17T20:19:39.2192451Z 2022-03-17T20:19:39.2192738Z Generating XML reports... 2022-03-17T20:19:39.2265010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201935.xml 2022-03-17T20:19:41.7889874Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:41.7903099Z 2022-03-17T20:19:41.7903356Z Running tests... 2022-03-17T20:19:41.7903790Z ---------------------------------------------------------------------- 2022-03-17T20:19:42.4148642Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94428 2022-03-17T20:19:42.4184417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94429 2022-03-17T20:19:42.4185991Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:19:43.5380192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:43.5407391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:43.5408362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:43.5482631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:43.5488698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:43.6415241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:43.9281382Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:44.0247736Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:45.7274625Z ok (3.936s) 2022-03-17T20:19:45.7275277Z 2022-03-17T20:19:45.7276545Z ---------------------------------------------------------------------- 2022-03-17T20:19:45.7277350Z Ran 1 test in 3.937s 2022-03-17T20:19:45.7277751Z 2022-03-17T20:19:45.7277943Z OK 2022-03-17T20:19:45.7278250Z 2022-03-17T20:19:45.7278537Z Generating XML reports... 2022-03-17T20:19:45.7356145Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201941.xml 2022-03-17T20:19:48.2945477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:48.2961115Z 2022-03-17T20:19:48.2961302Z Running tests... 2022-03-17T20:19:48.2961728Z ---------------------------------------------------------------------- 2022-03-17T20:19:48.9278025Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94643 2022-03-17T20:19:48.9314639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94644 2022-03-17T20:19:48.9315933Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:19:50.0462541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:50.0579773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:50.0580919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:50.0665835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:50.0671750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:50.1594556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:51.9413029Z skip (3.645s) 2022-03-17T20:19:51.9413532Z 2022-03-17T20:19:51.9414569Z ---------------------------------------------------------------------- 2022-03-17T20:19:51.9415530Z Ran 1 test in 3.645s 2022-03-17T20:19:51.9415985Z 2022-03-17T20:19:51.9416224Z OK (skipped=1) 2022-03-17T20:19:51.9416568Z 2022-03-17T20:19:51.9416836Z Generating XML reports... 2022-03-17T20:19:51.9498855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201948.xml 2022-03-17T20:19:54.5851209Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:54.5865567Z 2022-03-17T20:19:54.5865741Z Running tests... 2022-03-17T20:19:54.5866299Z ---------------------------------------------------------------------- 2022-03-17T20:19:55.1888796Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94851 2022-03-17T20:19:55.1921687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94852 2022-03-17T20:19:55.1923289Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:19:56.3153319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:56.3356085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:56.3356957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:56.3357772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:56.3371400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:56.4366933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:56.8204447Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:19:56.8857260Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:06.3017180Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:20:09.2160591Z ok (14.629s) 2022-03-17T20:20:09.2161069Z 2022-03-17T20:20:09.2162573Z ---------------------------------------------------------------------- 2022-03-17T20:20:09.2163675Z Ran 1 test in 14.629s 2022-03-17T20:20:09.2164042Z 2022-03-17T20:20:09.2164232Z OK 2022-03-17T20:20:09.2164523Z 2022-03-17T20:20:09.2164803Z Generating XML reports... 2022-03-17T20:20:09.2241334Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201954.xml 2022-03-17T20:20:11.7781637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:11.7795757Z 2022-03-17T20:20:11.7796016Z Running tests... 2022-03-17T20:20:11.7796446Z ---------------------------------------------------------------------- 2022-03-17T20:20:12.3824077Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95070 2022-03-17T20:20:12.3854473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95071 2022-03-17T20:20:12.3856059Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:20:13.4930545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:13.5146065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:13.5147067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:13.5232140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:13.5235484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:13.6159017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:13.9015827Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:13.9994196Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:23.3099748Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:20:23.3154753Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:20:26.3139262Z ok (14.534s) 2022-03-17T20:20:26.3139821Z 2022-03-17T20:20:26.3141360Z ---------------------------------------------------------------------- 2022-03-17T20:20:26.3142167Z Ran 1 test in 14.534s 2022-03-17T20:20:26.3142512Z 2022-03-17T20:20:26.3142728Z OK 2022-03-17T20:20:26.3143022Z 2022-03-17T20:20:26.3148125Z Generating XML reports... 2022-03-17T20:20:26.3218938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202011.xml 2022-03-17T20:20:28.9115199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:28.9126438Z 2022-03-17T20:20:28.9127190Z Running tests... 2022-03-17T20:20:28.9128649Z ---------------------------------------------------------------------- 2022-03-17T20:20:28.9149447Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:28.9150259Z 2022-03-17T20:20:28.9151305Z ---------------------------------------------------------------------- 2022-03-17T20:20:28.9152241Z Ran 1 test in 0.002s 2022-03-17T20:20:28.9152602Z 2022-03-17T20:20:28.9152834Z OK (skipped=1) 2022-03-17T20:20:28.9153197Z 2022-03-17T20:20:28.9153467Z Generating XML reports... 2022-03-17T20:20:28.9187618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202028.xml 2022-03-17T20:20:31.4550960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:31.4559357Z 2022-03-17T20:20:31.4559726Z Running tests... 2022-03-17T20:20:31.4561111Z ---------------------------------------------------------------------- 2022-03-17T20:20:32.0595387Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95356 2022-03-17T20:20:32.0634811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95357 2022-03-17T20:20:32.0636388Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:20:33.1707464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:33.1917075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:33.1917902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:33.2012173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:33.2018320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:33.2931199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:33.6779318Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:33.7310982Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:42.9272378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:42.9287720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:45.5888654Z ok (14.132s) 2022-03-17T20:20:45.5889100Z 2022-03-17T20:20:45.5890286Z ---------------------------------------------------------------------- 2022-03-17T20:20:45.5891247Z Ran 1 test in 14.133s 2022-03-17T20:20:45.5891667Z 2022-03-17T20:20:45.5891837Z OK 2022-03-17T20:20:45.5892128Z 2022-03-17T20:20:45.5892405Z Generating XML reports... 2022-03-17T20:20:45.5973562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202031.xml 2022-03-17T20:20:48.1629759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:48.1643969Z 2022-03-17T20:20:48.1644467Z Running tests... 2022-03-17T20:20:48.1645436Z ---------------------------------------------------------------------- 2022-03-17T20:20:48.8017672Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95575 2022-03-17T20:20:48.8065335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95576 2022-03-17T20:20:48.8066644Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:20:49.8946773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:49.9482485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:49.9484185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:49.9553158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:49.9554760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:50.0494557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:50.4387354Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:50.4875806Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:20:50.5928493Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:20:50.5931253Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:20:50.5954254Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:20:50.5955410Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:21:02.6353412Z ok (14.471s) 2022-03-17T20:21:02.6353820Z 2022-03-17T20:21:02.6354924Z ---------------------------------------------------------------------- 2022-03-17T20:21:02.6355721Z Ran 1 test in 14.471s 2022-03-17T20:21:02.6356104Z 2022-03-17T20:21:02.6356325Z OK 2022-03-17T20:21:02.6356645Z 2022-03-17T20:21:02.6356917Z Generating XML reports... 2022-03-17T20:21:02.6425979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202048.xml 2022-03-17T20:21:05.2111048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:05.2123985Z 2022-03-17T20:21:05.2124222Z Running tests... 2022-03-17T20:21:05.2124659Z ---------------------------------------------------------------------- 2022-03-17T20:21:05.8445626Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95856 2022-03-17T20:21:05.8483578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95857 2022-03-17T20:21:05.8484933Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:21:06.9345371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:06.9703216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:06.9705568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:06.9749416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:06.9751061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:07.0715372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:08.5701636Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:21:08.6268682Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:21:17.8896105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:17.8897543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:21.1737370Z ok (15.961s) 2022-03-17T20:21:21.1737941Z 2022-03-17T20:21:21.1739157Z ---------------------------------------------------------------------- 2022-03-17T20:21:21.1740121Z Ran 1 test in 15.961s 2022-03-17T20:21:21.1740587Z 2022-03-17T20:21:21.1740808Z OK 2022-03-17T20:21:21.1741725Z 2022-03-17T20:21:21.1747440Z Generating XML reports... 2022-03-17T20:21:21.1819515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202105.xml 2022-03-17T20:21:23.9149156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:23.9161290Z 2022-03-17T20:21:23.9161542Z Running tests... 2022-03-17T20:21:23.9162170Z ---------------------------------------------------------------------- 2022-03-17T20:21:24.5505929Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96075 2022-03-17T20:21:24.5548489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96076 2022-03-17T20:21:24.5549599Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:21:25.6234327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:25.6404298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:25.6406030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:25.6436807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:25.6446704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:25.7414761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:26.1649537Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:21:26.2038340Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:21:35.3832208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:35.3833497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:37.9779227Z ok (14.061s) 2022-03-17T20:21:37.9779613Z 2022-03-17T20:21:37.9781026Z ---------------------------------------------------------------------- 2022-03-17T20:21:37.9781797Z Ran 1 test in 14.061s 2022-03-17T20:21:37.9782157Z 2022-03-17T20:21:37.9782361Z OK 2022-03-17T20:21:37.9782648Z 2022-03-17T20:21:37.9783364Z Generating XML reports... 2022-03-17T20:21:37.9861951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202123.xml 2022-03-17T20:21:40.5960735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:40.5972329Z 2022-03-17T20:21:40.5972674Z Running tests... 2022-03-17T20:21:40.5973122Z ---------------------------------------------------------------------- 2022-03-17T20:21:41.2131361Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96294 2022-03-17T20:21:41.2176277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96295 2022-03-17T20:21:41.2177775Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:21:42.3634786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:42.3637581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:42.3639658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:42.3737084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:42.3741067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:42.4651722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:44.1276028Z skip (3.530s) 2022-03-17T20:21:44.1276423Z 2022-03-17T20:21:44.1277818Z ---------------------------------------------------------------------- 2022-03-17T20:21:44.1278573Z Ran 1 test in 3.530s 2022-03-17T20:21:44.1278955Z 2022-03-17T20:21:44.1279181Z OK (skipped=1) 2022-03-17T20:21:44.1279530Z 2022-03-17T20:21:44.1279822Z Generating XML reports... 2022-03-17T20:21:44.1349522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202140.xml 2022-03-17T20:21:46.8123019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:46.8136196Z 2022-03-17T20:21:46.8136748Z Running tests... 2022-03-17T20:21:46.8137910Z ---------------------------------------------------------------------- 2022-03-17T20:21:47.4206096Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96502 2022-03-17T20:21:47.4251582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96503 2022-03-17T20:21:47.4252968Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:21:48.5440539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:48.5468564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:48.5470499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:48.5542736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:48.5550437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:48.6479423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:50.4374729Z skip (3.624s) 2022-03-17T20:21:50.4375135Z 2022-03-17T20:21:50.4375947Z ---------------------------------------------------------------------- 2022-03-17T20:21:50.4376715Z Ran 1 test in 3.624s 2022-03-17T20:21:50.4377087Z 2022-03-17T20:21:50.4377850Z OK (skipped=1) 2022-03-17T20:21:50.4378546Z 2022-03-17T20:21:50.4378848Z Generating XML reports... 2022-03-17T20:21:50.4458313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202146.xml 2022-03-17T20:21:53.2264252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:53.2278703Z 2022-03-17T20:21:53.2278990Z Running tests... 2022-03-17T20:21:53.2279428Z ---------------------------------------------------------------------- 2022-03-17T20:21:53.8347276Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96710 2022-03-17T20:21:53.8390716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96711 2022-03-17T20:21:53.8392094Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:21:54.9686006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:54.9706507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:54.9708781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:54.9789139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:54.9791002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:55.0720805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:56.7487878Z skip (3.520s) 2022-03-17T20:21:56.7488509Z 2022-03-17T20:21:56.7489642Z ---------------------------------------------------------------------- 2022-03-17T20:21:56.7490502Z Ran 1 test in 3.521s 2022-03-17T20:21:56.7490893Z 2022-03-17T20:21:56.7491130Z OK (skipped=1) 2022-03-17T20:21:56.7491487Z 2022-03-17T20:21:56.7495634Z Generating XML reports... 2022-03-17T20:21:56.7569182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202153.xml 2022-03-17T20:21:59.3338339Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:59.3351486Z 2022-03-17T20:21:59.3352147Z Running tests... 2022-03-17T20:21:59.3353354Z ---------------------------------------------------------------------- 2022-03-17T20:21:59.9397278Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96918 2022-03-17T20:21:59.9433871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96919 2022-03-17T20:21:59.9435261Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:22:01.0227206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:01.0654776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:01.0656969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:01.0736326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:01.0742337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:01.1671288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:02.9531701Z skip (3.618s) 2022-03-17T20:22:02.9532074Z 2022-03-17T20:22:02.9534469Z ---------------------------------------------------------------------- 2022-03-17T20:22:02.9536145Z Ran 1 test in 3.618s 2022-03-17T20:22:02.9536624Z 2022-03-17T20:22:02.9536955Z OK (skipped=1) 2022-03-17T20:22:02.9537300Z 2022-03-17T20:22:02.9537570Z Generating XML reports... 2022-03-17T20:22:02.9615885Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202159.xml 2022-03-17T20:22:05.5468328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:05.5480884Z 2022-03-17T20:22:05.5481545Z Running tests... 2022-03-17T20:22:05.5483256Z ---------------------------------------------------------------------- 2022-03-17T20:22:05.5499979Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:05.5500941Z 2022-03-17T20:22:05.5501916Z ---------------------------------------------------------------------- 2022-03-17T20:22:05.5502849Z Ran 1 test in 0.002s 2022-03-17T20:22:05.5503317Z 2022-03-17T20:22:05.5503572Z OK (skipped=1) 2022-03-17T20:22:05.5503932Z 2022-03-17T20:22:05.5504188Z Generating XML reports... 2022-03-17T20:22:05.5542332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202205.xml 2022-03-17T20:22:08.5716646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:08.5727669Z 2022-03-17T20:22:08.5728248Z Running tests... 2022-03-17T20:22:08.5729679Z ---------------------------------------------------------------------- 2022-03-17T20:22:08.5746449Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:08.5747400Z 2022-03-17T20:22:08.5748384Z ---------------------------------------------------------------------- 2022-03-17T20:22:08.5749359Z Ran 1 test in 0.002s 2022-03-17T20:22:08.5749711Z 2022-03-17T20:22:08.5749948Z OK (skipped=1) 2022-03-17T20:22:08.5750298Z 2022-03-17T20:22:08.5750588Z Generating XML reports... 2022-03-17T20:22:08.5786686Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202208.xml 2022-03-17T20:22:11.2600601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:11.2612127Z 2022-03-17T20:22:11.2612735Z Running tests... 2022-03-17T20:22:11.2614164Z ---------------------------------------------------------------------- 2022-03-17T20:22:11.2632610Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:11.2633756Z 2022-03-17T20:22:11.2634743Z ---------------------------------------------------------------------- 2022-03-17T20:22:11.2635627Z Ran 1 test in 0.002s 2022-03-17T20:22:11.2635981Z 2022-03-17T20:22:11.2636218Z OK (skipped=1) 2022-03-17T20:22:11.2636567Z 2022-03-17T20:22:11.2636839Z Generating XML reports... 2022-03-17T20:22:11.2671008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202211.xml 2022-03-17T20:22:13.9261130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:13.9273580Z 2022-03-17T20:22:13.9274235Z Running tests... 2022-03-17T20:22:13.9275639Z ---------------------------------------------------------------------- 2022-03-17T20:22:13.9295071Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:13.9296209Z 2022-03-17T20:22:13.9297189Z ---------------------------------------------------------------------- 2022-03-17T20:22:13.9298088Z Ran 1 test in 0.002s 2022-03-17T20:22:13.9298463Z 2022-03-17T20:22:13.9298693Z OK (skipped=1) 2022-03-17T20:22:13.9299050Z 2022-03-17T20:22:13.9299332Z Generating XML reports... 2022-03-17T20:22:13.9333252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202213.xml 2022-03-17T20:22:16.5686817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:16.5699961Z 2022-03-17T20:22:16.5700257Z Running tests... 2022-03-17T20:22:16.5700695Z ---------------------------------------------------------------------- 2022-03-17T20:22:16.5719771Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:16.5720137Z 2022-03-17T20:22:16.5720518Z ---------------------------------------------------------------------- 2022-03-17T20:22:16.5720884Z Ran 1 test in 0.002s 2022-03-17T20:22:16.5721047Z 2022-03-17T20:22:16.5721151Z OK (skipped=1) 2022-03-17T20:22:16.5721313Z 2022-03-17T20:22:16.5721432Z Generating XML reports... 2022-03-17T20:22:16.5757225Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202216.xml 2022-03-17T20:22:19.2337516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:19.2349664Z 2022-03-17T20:22:19.2349900Z Running tests... 2022-03-17T20:22:19.2350899Z ---------------------------------------------------------------------- 2022-03-17T20:22:19.2370812Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:19.2372069Z 2022-03-17T20:22:19.2372874Z ---------------------------------------------------------------------- 2022-03-17T20:22:19.2373651Z Ran 1 test in 0.002s 2022-03-17T20:22:19.2374014Z 2022-03-17T20:22:19.2374245Z OK (skipped=1) 2022-03-17T20:22:19.2374597Z 2022-03-17T20:22:19.2374869Z Generating XML reports... 2022-03-17T20:22:19.2409397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202219.xml 2022-03-17T20:22:22.4499784Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:22.4512176Z 2022-03-17T20:22:22.4512756Z Running tests... 2022-03-17T20:22:22.4513921Z ---------------------------------------------------------------------- 2022-03-17T20:22:22.4533947Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:22.4534935Z 2022-03-17T20:22:22.4535735Z ---------------------------------------------------------------------- 2022-03-17T20:22:22.4536507Z Ran 1 test in 0.002s 2022-03-17T20:22:22.4536883Z 2022-03-17T20:22:22.4537111Z OK (skipped=1) 2022-03-17T20:22:22.4537463Z 2022-03-17T20:22:22.4537749Z Generating XML reports... 2022-03-17T20:22:22.4572731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202222.xml 2022-03-17T20:22:25.1925176Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:25.1936827Z 2022-03-17T20:22:25.1937390Z Running tests... 2022-03-17T20:22:25.1938566Z ---------------------------------------------------------------------- 2022-03-17T20:22:25.1957337Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:25.1958287Z 2022-03-17T20:22:25.1959108Z ---------------------------------------------------------------------- 2022-03-17T20:22:25.1959874Z Ran 1 test in 0.002s 2022-03-17T20:22:25.1960247Z 2022-03-17T20:22:25.1960484Z OK (skipped=1) 2022-03-17T20:22:25.1960855Z 2022-03-17T20:22:25.1961113Z Generating XML reports... 2022-03-17T20:22:25.1995632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202225.xml 2022-03-17T20:22:27.8772260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:27.8784502Z 2022-03-17T20:22:27.8784608Z Running tests... 2022-03-17T20:22:27.8785162Z ---------------------------------------------------------------------- 2022-03-17T20:22:27.8805693Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:27.8806609Z 2022-03-17T20:22:27.8807496Z ---------------------------------------------------------------------- 2022-03-17T20:22:27.8808289Z Ran 1 test in 0.002s 2022-03-17T20:22:27.8808638Z 2022-03-17T20:22:27.8808867Z OK (skipped=1) 2022-03-17T20:22:27.8809214Z 2022-03-17T20:22:27.8809482Z Generating XML reports... 2022-03-17T20:22:27.8845408Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202227.xml 2022-03-17T20:22:30.3696990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:30.3710405Z 2022-03-17T20:22:30.3710859Z Running tests... 2022-03-17T20:22:30.3712794Z ---------------------------------------------------------------------- 2022-03-17T20:22:30.3730466Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:30.3731256Z 2022-03-17T20:22:30.3732019Z ---------------------------------------------------------------------- 2022-03-17T20:22:30.3733151Z Ran 1 test in 0.002s 2022-03-17T20:22:30.3733517Z 2022-03-17T20:22:30.3733743Z OK (skipped=1) 2022-03-17T20:22:30.3734088Z 2022-03-17T20:22:30.3734348Z Generating XML reports... 2022-03-17T20:22:30.3772744Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202230.xml 2022-03-17T20:22:33.0479074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:33.0491367Z 2022-03-17T20:22:33.0492136Z Running tests... 2022-03-17T20:22:33.0493423Z ---------------------------------------------------------------------- 2022-03-17T20:22:33.0511876Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:33.0512766Z 2022-03-17T20:22:33.0513809Z ---------------------------------------------------------------------- 2022-03-17T20:22:33.0514830Z Ran 1 test in 0.002s 2022-03-17T20:22:33.0515231Z 2022-03-17T20:22:33.0515499Z OK (skipped=1) 2022-03-17T20:22:33.0515884Z 2022-03-17T20:22:33.0516163Z Generating XML reports... 2022-03-17T20:22:33.0551870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202233.xml 2022-03-17T20:22:35.7139270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:35.7151372Z 2022-03-17T20:22:35.7152050Z Running tests... 2022-03-17T20:22:35.7153032Z ---------------------------------------------------------------------- 2022-03-17T20:22:35.7170215Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:22:35.7170883Z 2022-03-17T20:22:35.7171569Z ---------------------------------------------------------------------- 2022-03-17T20:22:35.7172327Z Ran 1 test in 0.002s 2022-03-17T20:22:35.7172692Z 2022-03-17T20:22:35.7172907Z OK (skipped=1) 2022-03-17T20:22:35.7173248Z 2022-03-17T20:22:35.7173536Z Generating XML reports... 2022-03-17T20:22:35.7209129Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202235.xml 2022-03-17T20:22:38.7228266Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:38.7241176Z 2022-03-17T20:22:38.7241297Z Running tests... 2022-03-17T20:22:38.7241741Z ---------------------------------------------------------------------- 2022-03-17T20:22:39.3244104Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97930 2022-03-17T20:22:39.3285796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97931 2022-03-17T20:22:39.3287108Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:22:40.3993888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:40.4107513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:40.4109155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:40.4196893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:40.4198059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:40.5118744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:40.8012112Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:22:40.8933575Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:22:50.2021435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.2023104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.2235901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.2237167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.2365627Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:50.2369928Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:50.2842292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.2843581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.3003607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.3004852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.3104513Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:50.3106837Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:50.3110423Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:50.3112631Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:50.3176219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.3177505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.3335235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:50.3336508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:52.9569730Z ok (14.232s) 2022-03-17T20:22:52.9570129Z 2022-03-17T20:22:52.9572021Z ---------------------------------------------------------------------- 2022-03-17T20:22:52.9572908Z Ran 1 test in 14.233s 2022-03-17T20:22:52.9573313Z 2022-03-17T20:22:52.9573490Z OK 2022-03-17T20:22:52.9573800Z 2022-03-17T20:22:52.9574093Z Generating XML reports... 2022-03-17T20:22:52.9657734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202238.xml 2022-03-17T20:22:55.6711751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:55.6725546Z 2022-03-17T20:22:55.6725919Z Running tests... 2022-03-17T20:22:55.6727067Z ---------------------------------------------------------------------- 2022-03-17T20:22:56.2741570Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98149 2022-03-17T20:22:56.2775047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98150 2022-03-17T20:22:56.2776600Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:22:57.3930317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:57.4264469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:57.4266466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:57.4336547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:57.4343627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:57.5279454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:57.9114335Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:22:57.9626634Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:23:10.5061441Z ok (14.833s) 2022-03-17T20:23:10.5061918Z 2022-03-17T20:23:10.5063058Z ---------------------------------------------------------------------- 2022-03-17T20:23:10.5064026Z Ran 1 test in 14.834s 2022-03-17T20:23:10.5064457Z 2022-03-17T20:23:10.5064672Z OK 2022-03-17T20:23:10.5065046Z 2022-03-17T20:23:10.5065377Z Generating XML reports... 2022-03-17T20:23:10.5142959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202255.xml 2022-03-17T20:23:13.2438699Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:13.2453399Z 2022-03-17T20:23:13.2453962Z Running tests... 2022-03-17T20:23:13.2455186Z ---------------------------------------------------------------------- 2022-03-17T20:23:13.8632584Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98420 2022-03-17T20:23:13.8677304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98421 2022-03-17T20:23:13.8678644Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:23:14.9840053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:15.0250499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:15.0252281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:15.0352026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:15.0362396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:15.1262848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:24.4219157Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:23:24.5781802Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:23:24.7338126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:24.7339399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:27.3968884Z ok (14.151s) 2022-03-17T20:23:27.3969399Z 2022-03-17T20:23:27.3970592Z ---------------------------------------------------------------------- 2022-03-17T20:23:27.3971629Z Ran 1 test in 14.152s 2022-03-17T20:23:27.3972004Z 2022-03-17T20:23:27.3972173Z OK 2022-03-17T20:23:27.3972463Z 2022-03-17T20:23:27.3972756Z Generating XML reports... 2022-03-17T20:23:27.4056084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202313.xml 2022-03-17T20:23:30.0904379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:30.0916651Z 2022-03-17T20:23:30.0916986Z Running tests... 2022-03-17T20:23:30.0918139Z ---------------------------------------------------------------------- 2022-03-17T20:23:30.0996529Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.008s) 2022-03-17T20:23:30.0997381Z 2022-03-17T20:23:30.0998643Z ---------------------------------------------------------------------- 2022-03-17T20:23:30.0999852Z Ran 1 test in 0.008s 2022-03-17T20:23:30.1000409Z 2022-03-17T20:23:30.1000730Z OK (skipped=1) 2022-03-17T20:23:30.1001181Z 2022-03-17T20:23:30.1001500Z Generating XML reports... 2022-03-17T20:23:30.1036176Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202330.xml 2022-03-17T20:23:32.8856902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:32.8869126Z 2022-03-17T20:23:32.8869541Z Running tests... 2022-03-17T20:23:32.8870910Z ---------------------------------------------------------------------- 2022-03-17T20:23:33.4885913Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98706 2022-03-17T20:23:33.4928167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98707 2022-03-17T20:23:33.4929761Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:23:34.5614426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:34.6221737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:34.6222813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:34.6226449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:34.6237817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:34.7230712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:35.0990501Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:23:35.1653547Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:23:44.5200375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:44.5202309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:47.2255915Z ok (14.338s) 2022-03-17T20:23:47.2256467Z 2022-03-17T20:23:47.2258053Z ---------------------------------------------------------------------- 2022-03-17T20:23:47.2258995Z Ran 1 test in 14.338s 2022-03-17T20:23:47.2259445Z 2022-03-17T20:23:47.2259677Z OK 2022-03-17T20:23:47.2260017Z 2022-03-17T20:23:47.2260344Z Generating XML reports... 2022-03-17T20:23:47.2341137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202332.xml 2022-03-17T20:23:49.8892035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:49.8903778Z 2022-03-17T20:23:49.8904282Z Running tests... 2022-03-17T20:23:49.8905654Z ---------------------------------------------------------------------- 2022-03-17T20:23:50.4954196Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98925 2022-03-17T20:23:50.4991276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98926 2022-03-17T20:23:50.4992608Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:23:51.5658827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:51.6190918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:51.6191677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:51.6264992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:51.6269595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:51.7203026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:53.5086925Z skip (3.618s) 2022-03-17T20:23:53.5087391Z 2022-03-17T20:23:53.5088445Z ---------------------------------------------------------------------- 2022-03-17T20:23:53.5089227Z Ran 1 test in 3.618s 2022-03-17T20:23:53.5090616Z 2022-03-17T20:23:53.5091205Z OK (skipped=1) 2022-03-17T20:23:53.5091689Z 2022-03-17T20:23:53.5092049Z Generating XML reports... 2022-03-17T20:23:53.5174658Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202349.xml 2022-03-17T20:23:56.2981509Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:56.2994143Z 2022-03-17T20:23:56.2995050Z Running tests... 2022-03-17T20:23:56.2996475Z ---------------------------------------------------------------------- 2022-03-17T20:23:56.9125820Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99133 2022-03-17T20:23:56.9170600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99134 2022-03-17T20:23:56.9172972Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:23:58.0069230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:58.1062365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:58.1064458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:58.1078518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:58.1082293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:58.2076418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:24:00.3277788Z skip (4.028s) 2022-03-17T20:24:00.3278353Z 2022-03-17T20:24:00.3279555Z ---------------------------------------------------------------------- 2022-03-17T20:24:00.3280441Z Ran 1 test in 4.028s 2022-03-17T20:24:00.3280872Z 2022-03-17T20:24:00.3281163Z OK (skipped=1) 2022-03-17T20:24:00.3281565Z 2022-03-17T20:24:00.3282213Z Generating XML reports... 2022-03-17T20:24:00.3351037Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202356.xml 2022-03-17T20:24:02.9485394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:24:02.9498350Z 2022-03-17T20:24:02.9499157Z Running tests... 2022-03-17T20:24:02.9500438Z ---------------------------------------------------------------------- 2022-03-17T20:24:03.5558283Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99341 2022-03-17T20:24:03.5597156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99342 2022-03-17T20:24:03.5598745Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:24:04.6683307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:24:04.7151367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:24:04.7153083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:04.7189020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:04.7192253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:24:04.8170620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:24:05.2063109Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:05.2661608Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:18.2860146Z ok (15.336s) 2022-03-17T20:24:18.2860539Z 2022-03-17T20:24:18.2861560Z ---------------------------------------------------------------------- 2022-03-17T20:24:18.2862389Z Ran 1 test in 15.336s 2022-03-17T20:24:18.2862763Z 2022-03-17T20:24:18.2863339Z OK 2022-03-17T20:24:18.2863682Z 2022-03-17T20:24:18.2863986Z Generating XML reports... 2022-03-17T20:24:18.2941252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202402.xml 2022-03-17T20:24:20.9440832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:24:20.9451870Z 2022-03-17T20:24:20.9452119Z Running tests... 2022-03-17T20:24:20.9452987Z ---------------------------------------------------------------------- 2022-03-17T20:24:21.5483497Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99775 2022-03-17T20:24:21.5521296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99776 2022-03-17T20:24:21.5522823Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:24:22.6188861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:24:22.6600174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:24:22.6602310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:22.6699197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:22.6705090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:24:22.7616219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:24:23.0595153Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:23.1680880Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:35.3802080Z ok (14.434s) 2022-03-17T20:24:35.3803145Z 2022-03-17T20:24:35.3804253Z ---------------------------------------------------------------------- 2022-03-17T20:24:35.3805153Z Ran 1 test in 14.435s 2022-03-17T20:24:35.3805573Z 2022-03-17T20:24:35.3805780Z OK 2022-03-17T20:24:35.3806123Z 2022-03-17T20:24:35.3806447Z Generating XML reports... 2022-03-17T20:24:35.3884472Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202420.xml 2022-03-17T20:24:37.9402341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:24:37.9412300Z 2022-03-17T20:24:37.9412628Z Running tests... 2022-03-17T20:24:37.9413180Z ---------------------------------------------------------------------- 2022-03-17T20:24:38.5444857Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100002 2022-03-17T20:24:38.5482686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100003 2022-03-17T20:24:38.5484047Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:24:39.6535677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:24:39.6756874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:24:39.6757827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:39.6838186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:39.6840277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:24:39.7767135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:24:40.1587803Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:40.2048166Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:52.1725648Z ok (14.231s) 2022-03-17T20:24:52.1726084Z 2022-03-17T20:24:52.1727111Z ---------------------------------------------------------------------- 2022-03-17T20:24:52.1727883Z Ran 1 test in 14.231s 2022-03-17T20:24:52.1728257Z 2022-03-17T20:24:52.1728450Z OK 2022-03-17T20:24:52.1734238Z 2022-03-17T20:24:52.1736162Z Generating XML reports... 2022-03-17T20:24:52.1821296Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202437.xml 2022-03-17T20:24:55.1026346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:24:55.1036855Z 2022-03-17T20:24:55.1037392Z Running tests... 2022-03-17T20:24:55.1038641Z ---------------------------------------------------------------------- 2022-03-17T20:24:55.7068908Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100217 2022-03-17T20:24:55.7102944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100218 2022-03-17T20:24:55.7104361Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:24:56.7778334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:24:56.8284959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:24:56.8287136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:56.8381736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:24:56.8383795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:24:56.9295187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:24:57.3142113Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:24:57.3673374Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:06.3752132Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:25:06.4112683Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:25:06.5704185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:06.5715506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:09.2384887Z ok (14.134s) 2022-03-17T20:25:09.2385525Z 2022-03-17T20:25:09.2387032Z ---------------------------------------------------------------------- 2022-03-17T20:25:09.2388164Z Ran 1 test in 14.135s 2022-03-17T20:25:09.2388699Z 2022-03-17T20:25:09.2388961Z OK 2022-03-17T20:25:09.2389419Z 2022-03-17T20:25:09.2389787Z Generating XML reports... 2022-03-17T20:25:09.2462068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202455.xml 2022-03-17T20:25:11.9648936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:11.9659710Z 2022-03-17T20:25:11.9660232Z Running tests... 2022-03-17T20:25:11.9661903Z ---------------------------------------------------------------------- 2022-03-17T20:25:12.5706467Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100436 2022-03-17T20:25:12.5746075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100437 2022-03-17T20:25:12.5747373Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:25:13.6411949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:13.6516594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:13.6517495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:13.6614981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:13.6615570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:13.7529890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:14.1331119Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:14.1910138Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:14.2884377Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:25:14.2885792Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:25:14.2920847Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:25:14.2921972Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:25:26.2060008Z ok (14.239s) 2022-03-17T20:25:26.2060485Z 2022-03-17T20:25:26.2061561Z ---------------------------------------------------------------------- 2022-03-17T20:25:26.2062417Z Ran 1 test in 14.240s 2022-03-17T20:25:26.2062767Z 2022-03-17T20:25:26.2062966Z OK 2022-03-17T20:25:26.2063266Z 2022-03-17T20:25:26.2063561Z Generating XML reports... 2022-03-17T20:25:26.2142288Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202511.xml 2022-03-17T20:25:29.0831537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:29.0843149Z 2022-03-17T20:25:29.0843973Z Running tests... 2022-03-17T20:25:29.0845121Z ---------------------------------------------------------------------- 2022-03-17T20:25:29.6888933Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100655 2022-03-17T20:25:29.6929151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100656 2022-03-17T20:25:29.6930523Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:25:30.7665088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:30.7722376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:30.7724686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:30.7765753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:30.7767083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:30.8731064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:31.1566880Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:31.2525957Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:40.4005375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:40.4015820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:40.4562569Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:25:40.4566599Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:25:43.1165160Z ok (14.032s) 2022-03-17T20:25:43.1165574Z 2022-03-17T20:25:43.1166609Z ---------------------------------------------------------------------- 2022-03-17T20:25:43.1167368Z Ran 1 test in 14.032s 2022-03-17T20:25:43.1167725Z 2022-03-17T20:25:43.1169859Z OK 2022-03-17T20:25:43.1170296Z 2022-03-17T20:25:43.1170591Z Generating XML reports... 2022-03-17T20:25:43.1250225Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202529.xml 2022-03-17T20:25:45.7215868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:45.7231502Z 2022-03-17T20:25:45.7231911Z Running tests... 2022-03-17T20:25:45.7232454Z ---------------------------------------------------------------------- 2022-03-17T20:25:46.3167025Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.594s) 2022-03-17T20:25:46.3167346Z 2022-03-17T20:25:46.3167736Z ---------------------------------------------------------------------- 2022-03-17T20:25:46.3168084Z Ran 1 test in 0.594s 2022-03-17T20:25:46.3168260Z 2022-03-17T20:25:46.3168373Z OK (skipped=1) 2022-03-17T20:25:46.3168538Z 2022-03-17T20:25:46.3168649Z Generating XML reports... 2022-03-17T20:25:46.3196872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202545.xml 2022-03-17T20:25:48.9316299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:48.9330475Z 2022-03-17T20:25:48.9331013Z Running tests... 2022-03-17T20:25:48.9332300Z ---------------------------------------------------------------------- 2022-03-17T20:25:49.5346545Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100947 2022-03-17T20:25:49.5382423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100948 2022-03-17T20:25:49.5384631Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:25:50.6782749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:50.7004345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:50.7006262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:50.7085469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:50.7087297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:50.8017488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:51.1877523Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:51.2409591Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:53.4481771Z ok (4.515s) 2022-03-17T20:25:53.4482423Z 2022-03-17T20:25:53.4483698Z ---------------------------------------------------------------------- 2022-03-17T20:25:53.4484693Z Ran 1 test in 4.515s 2022-03-17T20:25:53.4485113Z 2022-03-17T20:25:53.4485349Z OK 2022-03-17T20:25:53.4485708Z 2022-03-17T20:25:53.4486084Z Generating XML reports... 2022-03-17T20:25:53.4566396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202548.xml 2022-03-17T20:25:56.3095595Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:56.3107954Z 2022-03-17T20:25:56.3108759Z Running tests... 2022-03-17T20:25:56.3109575Z ---------------------------------------------------------------------- 2022-03-17T20:25:56.9159089Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101162 2022-03-17T20:25:56.9198615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101163 2022-03-17T20:25:56.9199946Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:25:58.0459451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:58.0476874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:58.0477777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:58.0561708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:58.0562615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:58.1483972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:58.5382305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:58.5999446Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:25:58.7049645Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:25:58.7051934Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:25:58.7055573Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:25:58.7057701Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:26:07.9341483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:07.9342772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:07.9427844Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:26:07.9436662Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:26:07.9599564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:07.9600763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:07.9698188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:07.9699425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:10.6462034Z ok (14.335s) 2022-03-17T20:26:10.6462510Z 2022-03-17T20:26:10.6463545Z ---------------------------------------------------------------------- 2022-03-17T20:26:10.6464332Z Ran 1 test in 14.335s 2022-03-17T20:26:10.6464698Z 2022-03-17T20:26:10.6465714Z OK 2022-03-17T20:26:10.6465994Z 2022-03-17T20:26:10.6466296Z Generating XML reports... 2022-03-17T20:26:10.6544748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202556.xml 2022-03-17T20:26:13.4639370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:13.4649871Z 2022-03-17T20:26:13.4650269Z Running tests... 2022-03-17T20:26:13.4650876Z ---------------------------------------------------------------------- 2022-03-17T20:26:14.0721736Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101381 2022-03-17T20:26:14.0758424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101382 2022-03-17T20:26:14.0759791Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:26:15.1849743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:15.2011660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:15.2013569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:15.2054334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:15.2056876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:15.3025264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:15.6841014Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:15.7354379Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:15.8519490Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:26:15.8521707Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:26:15.8550025Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:26:15.8552191Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:26:25.0146389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:25.0154531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:27.7026753Z ok (14.237s) 2022-03-17T20:26:27.7027153Z 2022-03-17T20:26:27.7028126Z ---------------------------------------------------------------------- 2022-03-17T20:26:27.7028912Z Ran 1 test in 14.237s 2022-03-17T20:26:27.7029277Z 2022-03-17T20:26:27.7029467Z OK 2022-03-17T20:26:27.7029758Z 2022-03-17T20:26:27.7030046Z Generating XML reports... 2022-03-17T20:26:27.7102515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202613.xml 2022-03-17T20:26:30.3294878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:30.3306966Z 2022-03-17T20:26:30.3307277Z Running tests... 2022-03-17T20:26:30.3307821Z ---------------------------------------------------------------------- 2022-03-17T20:26:30.9360346Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101600 2022-03-17T20:26:30.9393820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101601 2022-03-17T20:26:30.9395306Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:26:32.0329578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:32.0386841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:32.0388900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:32.0434732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:32.0438893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:32.1396050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:32.4222523Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:32.5163607Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:41.6617176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:41.6623974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:41.7458504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:41.7459731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:44.5736095Z ok (14.242s) 2022-03-17T20:26:44.5736496Z 2022-03-17T20:26:44.5738350Z ---------------------------------------------------------------------- 2022-03-17T20:26:44.5739228Z Ran 1 test in 14.243s 2022-03-17T20:26:44.5739595Z 2022-03-17T20:26:44.5739791Z OK 2022-03-17T20:26:44.5740102Z 2022-03-17T20:26:44.5740403Z Generating XML reports... 2022-03-17T20:26:44.5818192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202630.xml 2022-03-17T20:26:47.4223564Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:47.4233480Z 2022-03-17T20:26:47.4234025Z Running tests... 2022-03-17T20:26:47.4235160Z ---------------------------------------------------------------------- 2022-03-17T20:26:48.0412220Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... skip (0.617s) 2022-03-17T20:26:48.0413589Z 2022-03-17T20:26:48.0415194Z ---------------------------------------------------------------------- 2022-03-17T20:26:48.0416140Z Ran 1 test in 0.617s 2022-03-17T20:26:48.0416515Z 2022-03-17T20:26:48.0416755Z OK (skipped=1) 2022-03-17T20:26:48.0417122Z 2022-03-17T20:26:48.0418038Z Generating XML reports... 2022-03-17T20:26:48.0443926Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202647.xml 2022-03-17T20:26:50.6409785Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:50.6419623Z 2022-03-17T20:26:50.6419894Z Running tests... 2022-03-17T20:26:50.6420426Z ---------------------------------------------------------------------- 2022-03-17T20:26:51.2764270Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101886 2022-03-17T20:26:51.2799553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101887 2022-03-17T20:26:51.2800162Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:26:52.3997168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:52.4181605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:52.4183641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:52.4199393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:52.4204331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:52.5196196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:52.9030733Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:52.9518536Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:54.9892424Z ok (4.346s) 2022-03-17T20:26:54.9892831Z 2022-03-17T20:26:54.9893979Z ---------------------------------------------------------------------- 2022-03-17T20:26:54.9894972Z Ran 1 test in 4.347s 2022-03-17T20:26:54.9895388Z 2022-03-17T20:26:54.9895589Z OK 2022-03-17T20:26:54.9895879Z 2022-03-17T20:26:54.9896183Z Generating XML reports... 2022-03-17T20:26:54.9978967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202650.xml 2022-03-17T20:26:57.5822427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:57.5835321Z 2022-03-17T20:26:57.5835476Z Running tests... 2022-03-17T20:26:57.5835996Z ---------------------------------------------------------------------- 2022-03-17T20:26:58.1875406Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102101 2022-03-17T20:26:58.1909869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102102 2022-03-17T20:26:58.1911237Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:26:59.3061387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:59.3269383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:59.3270343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:59.3366116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:59.3372436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:59.4279399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:59.8105346Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:26:59.8638604Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:27:02.1027504Z ok (4.519s) 2022-03-17T20:27:02.1027895Z 2022-03-17T20:27:02.1028933Z ---------------------------------------------------------------------- 2022-03-17T20:27:02.1029685Z Ran 1 test in 4.519s 2022-03-17T20:27:02.1030066Z 2022-03-17T20:27:02.1030255Z OK 2022-03-17T20:27:02.1030544Z 2022-03-17T20:27:02.1030804Z Generating XML reports... 2022-03-17T20:27:02.1105465Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202657.xml 2022-03-17T20:27:04.6919575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:04.6931006Z 2022-03-17T20:27:04.6931540Z Running tests... 2022-03-17T20:27:04.6932705Z ---------------------------------------------------------------------- 2022-03-17T20:27:05.3008147Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102316 2022-03-17T20:27:05.3086676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102317 2022-03-17T20:27:05.3088092Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:27:06.4186291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:27:06.4408115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:27:06.4410028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:06.4493844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:06.4500132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:27:06.5417801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:27:06.9300174Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:27:06.9828531Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:27:16.0915617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:16.0922495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:18.7345479Z ok (14.041s) 2022-03-17T20:27:18.7345883Z 2022-03-17T20:27:18.7347059Z ---------------------------------------------------------------------- 2022-03-17T20:27:18.7348102Z Ran 1 test in 14.042s 2022-03-17T20:27:18.7348578Z 2022-03-17T20:27:18.7348825Z OK 2022-03-17T20:27:18.7349210Z 2022-03-17T20:27:18.7349506Z Generating XML reports... 2022-03-17T20:27:18.7429938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202704.xml 2022-03-17T20:27:21.7131772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:21.7140923Z 2022-03-17T20:27:21.7141212Z Running tests... 2022-03-17T20:27:21.7142318Z ---------------------------------------------------------------------- 2022-03-17T20:27:22.3477453Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102535 2022-03-17T20:27:22.3525539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102536 2022-03-17T20:27:22.3526884Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:27:23.5083400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:27:23.5240929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:27:23.5243843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:23.5286928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:23.5296128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:27:23.6257851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:27:23.8854809Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:27:23.8856509Z Running test: batch_norm_net sync interval 2022-03-17T20:27:23.8857267Z 1 with iteration mapping 2022-03-17T20:27:23.8857894Z {0: 0, 1: 2} 2022-03-17T20:27:23.9307915Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:27:23.9813151Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:27:24.0294745Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:27:34.2796446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:34.2877215Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:34.2879523Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:34.2883145Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:34.2885231Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:34.4340995Z Running test: batch_norm_net sync interval 2022-03-17T20:27:34.4341752Z 1 with iteration mapping 2022-03-17T20:27:34.4342391Z {0: 0, 1: 3} 2022-03-17T20:27:34.4397429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:34.4460672Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:34.4462723Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:34.4465517Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:34.4467451Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:34.4554482Z Running test: batch_norm_net sync interval 2022-03-17T20:27:34.4555587Z 1 with iteration mapping 2022-03-17T20:27:34.4556209Z {0: 0, 1: 10} 2022-03-17T20:27:34.4611183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:34.4884284Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:34.4886401Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:34.4889365Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:34.4891422Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:34.4938602Z Running test: batch_norm_net sync interval 2022-03-17T20:27:34.4939652Z 1 with iteration mapping 2022-03-17T20:27:34.4940278Z {0: 5, 1: 7} 2022-03-17T20:27:43.3927219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:43.3931624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:43.4106871Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:43.4108951Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:43.4111832Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:43.4114129Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:43.4184874Z Running test: batch_norm_net sync interval 2022-03-17T20:27:43.4185913Z 1 with iteration mapping 2022-03-17T20:27:43.4186732Z {0: 5, 1: 8} 2022-03-17T20:27:43.4242359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:43.4243551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:43.4454528Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:43.4456627Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:43.4459459Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:43.4461417Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:43.4528455Z Running test: batch_norm_net sync interval 2022-03-17T20:27:43.4529454Z 1 with iteration mapping 2022-03-17T20:27:43.4530088Z {0: 5, 1: 15} 2022-03-17T20:27:43.4576492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:43.4577725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:27:43.4948890Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:43.4951023Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:43.4953907Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:27:43.4955963Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:27:43.5034479Z Running test: large_conv_model sync interval 2022-03-17T20:27:43.5035619Z 1 with iteration mapping 2022-03-17T20:27:43.5036302Z {0: 0, 1: 2} 2022-03-17T20:28:10.5043388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:10.6895460Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:10.6898676Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:10.6902167Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:10.6904263Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:10.6959923Z Running test: large_conv_model sync interval 2022-03-17T20:28:10.6960782Z 1 with iteration mapping 2022-03-17T20:28:10.6961441Z {0: 0, 1: 3} 2022-03-17T20:28:10.8818791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:11.2487401Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:11.2489477Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:11.2492332Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:11.2494285Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:11.2541549Z Running test: large_conv_model sync interval 2022-03-17T20:28:11.2542830Z 1 with iteration mapping 2022-03-17T20:28:11.2543457Z {0: 0, 1: 10} 2022-03-17T20:28:11.4388535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:13.0823840Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:13.0825904Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:13.0828681Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:13.0830637Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:13.0880025Z Running test: large_conv_model sync interval 2022-03-17T20:28:13.0880821Z 1 with iteration mapping 2022-03-17T20:28:13.0881432Z {0: 5, 1: 7} 2022-03-17T20:28:13.4053090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:13.4054286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:14.5011802Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:14.5013806Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:14.5016626Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:14.5018934Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:14.5070531Z Running test: large_conv_model sync interval 2022-03-17T20:28:14.5071329Z 1 with iteration mapping 2022-03-17T20:28:14.5071940Z {0: 5, 1: 8} 2022-03-17T20:28:14.6916165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:14.6917388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:15.9700491Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:15.9702532Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:15.9705342Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:15.9707296Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:15.9760912Z Running test: large_conv_model sync interval 2022-03-17T20:28:15.9762944Z 1 with iteration mapping 2022-03-17T20:28:15.9763585Z {0: 5, 1: 15} 2022-03-17T20:28:16.1609236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:16.1610817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7160848Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.7163092Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.7165907Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.7167863Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.7219044Z Running test: small_model sync interval 2022-03-17T20:28:18.7219846Z 1 with iteration mapping 2022-03-17T20:28:18.7220498Z {0: 0, 1: 2} 2022-03-17T20:28:18.7296848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7345012Z Running test: small_model sync interval 2022-03-17T20:28:18.7346026Z 1 with iteration mapping 2022-03-17T20:28:18.7346874Z {0: 0, 1: 3} 2022-03-17T20:28:18.7379453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7447012Z Running test: small_model sync interval 2022-03-17T20:28:18.7447934Z 1 with iteration mapping 2022-03-17T20:28:18.7448806Z {0: 0, 1: 10} 2022-03-17T20:28:18.7483755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7650391Z Running test: small_model sync interval 2022-03-17T20:28:18.7651413Z 1 with iteration mapping 2022-03-17T20:28:18.7652277Z {0: 5, 1: 7} 2022-03-17T20:28:18.7689301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7690822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7825448Z Running test: small_model sync interval 2022-03-17T20:28:18.7826460Z 1 with iteration mapping 2022-03-17T20:28:18.7827301Z {0: 5, 1: 8} 2022-03-17T20:28:18.7874900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.7876515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.8024971Z Running test: small_model sync interval 2022-03-17T20:28:18.8026011Z 1 with iteration mapping 2022-03-17T20:28:18.8026865Z {0: 5, 1: 15} 2022-03-17T20:28:18.8066231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.8067833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:18.8311093Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.8312259Z 1 with iteration mapping 2022-03-17T20:28:18.8313017Z {0: 0, 1: 2} 2022-03-17T20:28:18.8365539Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8368056Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8371029Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8373071Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8385066Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.8386203Z 1 with iteration mapping 2022-03-17T20:28:18.8386965Z {0: 0, 1: 3} 2022-03-17T20:28:18.8440267Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8442477Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8445322Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8447268Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8455462Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.8456407Z 1 with iteration mapping 2022-03-17T20:28:18.8457048Z {0: 0, 1: 10} 2022-03-17T20:28:18.8581219Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8583004Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8585421Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8587074Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8611679Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.8612627Z 1 with iteration mapping 2022-03-17T20:28:18.8613228Z {0: 5, 1: 7} 2022-03-17T20:28:18.8642090Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:28:18.8748189Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8750193Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8752603Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8754269Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8775903Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.8776840Z 1 with iteration mapping 2022-03-17T20:28:18.8777502Z {0: 5, 1: 8} 2022-03-17T20:28:18.8939161Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8941231Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8944040Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.8945869Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.8968910Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.8969857Z 1 with iteration mapping 2022-03-17T20:28:18.8970620Z {0: 5, 1: 15} 2022-03-17T20:28:18.9217719Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9219898Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9222885Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9224925Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9248040Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.9249225Z 1 with iteration mapping 2022-03-17T20:28:18.9250020Z {0: 0, 1: 2} 2022-03-17T20:28:18.9272827Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:28:18.9299708Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9301400Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9303420Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9304809Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9318409Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.9319235Z 1 with iteration mapping 2022-03-17T20:28:18.9319797Z {0: 0, 1: 3} 2022-03-17T20:28:18.9374379Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9376031Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9378236Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9379806Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9391644Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.9392457Z 1 with iteration mapping 2022-03-17T20:28:18.9393057Z {0: 0, 1: 10} 2022-03-17T20:28:18.9519444Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9521065Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9523606Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9525164Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9548583Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.9549577Z 1 with iteration mapping 2022-03-17T20:28:18.9550238Z {0: 5, 1: 7} 2022-03-17T20:28:18.9674496Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9676517Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9679318Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9681599Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9707624Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.9708532Z 1 with iteration mapping 2022-03-17T20:28:18.9709146Z {0: 5, 1: 8} 2022-03-17T20:28:18.9844398Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9846407Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9849182Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:18.9851115Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:18.9879989Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:18.9880710Z 1 with iteration mapping 2022-03-17T20:28:18.9881248Z {0: 5, 1: 15} 2022-03-17T20:28:19.0101916Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.0103931Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.0107006Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.0108965Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.0120954Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:19.0121690Z 1 with iteration mapping 2022-03-17T20:28:19.0122470Z {0: 0, 1: 2} 2022-03-17T20:28:19.0140974Z registered hook 2022-03-17T20:28:19.0141806Z registered hook 2022-03-17T20:28:19.0168020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0224105Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:19.0224810Z 1 with iteration mapping 2022-03-17T20:28:19.0225334Z {0: 0, 1: 3} 2022-03-17T20:28:19.0231653Z registered hook 2022-03-17T20:28:19.0232565Z registered hook 2022-03-17T20:28:19.0259769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0328176Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:19.0329327Z 1 with iteration mapping 2022-03-17T20:28:19.0329942Z {0: 0, 1: 10} 2022-03-17T20:28:19.0338388Z registered hook 2022-03-17T20:28:19.0339430Z registered hook 2022-03-17T20:28:19.0363812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0549763Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:19.0550837Z 1 with iteration mapping 2022-03-17T20:28:19.0551603Z {0: 5, 1: 7} 2022-03-17T20:28:19.0564136Z registered hook 2022-03-17T20:28:19.0565360Z registered hook 2022-03-17T20:28:19.0595065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0596631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0737200Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:19.0738348Z 1 with iteration mapping 2022-03-17T20:28:19.0739303Z {0: 5, 1: 8} 2022-03-17T20:28:19.0744854Z registered hook 2022-03-17T20:28:19.0746056Z registered hook 2022-03-17T20:28:19.0774389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0776021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0937489Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:19.0938653Z 1 with iteration mapping 2022-03-17T20:28:19.0939509Z {0: 5, 1: 15} 2022-03-17T20:28:19.0943997Z registered hook 2022-03-17T20:28:19.0945210Z registered hook 2022-03-17T20:28:19.0975354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.0976987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.1244934Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:19.1246035Z 1 with iteration mapping 2022-03-17T20:28:19.1246841Z {0: 0, 1: 2} 2022-03-17T20:28:19.1255383Z registered hook 2022-03-17T20:28:19.1256971Z registered hook 2022-03-17T20:28:19.1261947Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:28:19.1267099Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:28:19.1285467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.1287423Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:28:19.1289691Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:28:19.1689631Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:19.1690755Z 1 with iteration mapping 2022-03-17T20:28:19.1691558Z {0: 0, 1: 3} 2022-03-17T20:28:19.1699922Z registered hook 2022-03-17T20:28:19.1701086Z registered hook 2022-03-17T20:28:19.1740040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.1828746Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:19.1829669Z 1 with iteration mapping 2022-03-17T20:28:19.1830779Z {0: 0, 1: 10} 2022-03-17T20:28:19.1841700Z registered hook 2022-03-17T20:28:19.1842689Z registered hook 2022-03-17T20:28:19.1880679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.2135209Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:19.2136183Z 1 with iteration mapping 2022-03-17T20:28:19.2137016Z {0: 5, 1: 7} 2022-03-17T20:28:19.2148096Z registered hook 2022-03-17T20:28:19.2149259Z registered hook 2022-03-17T20:28:19.2194933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.2196223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.2406106Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:19.2407245Z 1 with iteration mapping 2022-03-17T20:28:19.2408074Z {0: 5, 1: 8} 2022-03-17T20:28:19.2418209Z registered hook 2022-03-17T20:28:19.2419466Z registered hook 2022-03-17T20:28:19.2459109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.2460734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.2681403Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:19.2682479Z 1 with iteration mapping 2022-03-17T20:28:19.2683236Z {0: 5, 1: 15} 2022-03-17T20:28:19.2693495Z registered hook 2022-03-17T20:28:19.2696554Z registered hook 2022-03-17T20:28:19.2739993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.2741336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3139352Z Running test: batch_norm_net sync interval 2022-03-17T20:28:19.3140390Z 2 with iteration mapping 2022-03-17T20:28:19.3141254Z {0: 0, 1: 2} 2022-03-17T20:28:19.3203428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3239065Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3241237Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3244405Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3246433Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3294533Z Running test: batch_norm_net sync interval 2022-03-17T20:28:19.3295524Z 2 with iteration mapping 2022-03-17T20:28:19.3296167Z {0: 0, 1: 3} 2022-03-17T20:28:19.3335716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3377333Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3379806Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3382820Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3384882Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3431632Z Running test: batch_norm_net sync interval 2022-03-17T20:28:19.3432702Z 2 with iteration mapping 2022-03-17T20:28:19.3433550Z {0: 0, 1: 10} 2022-03-17T20:28:19.3470897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3608659Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3610445Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3612814Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3614451Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3666720Z Running test: batch_norm_net sync interval 2022-03-17T20:28:19.3667575Z 2 with iteration mapping 2022-03-17T20:28:19.3668209Z {0: 5, 1: 7} 2022-03-17T20:28:19.3713304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3714575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3825196Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3826969Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3829363Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.3831020Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.3916175Z Running test: batch_norm_net sync interval 2022-03-17T20:28:19.3917042Z 2 with iteration mapping 2022-03-17T20:28:19.3917715Z {0: 5, 1: 8} 2022-03-17T20:28:19.3961385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.3962867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.4085322Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.4087289Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.4089700Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.4091329Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.4141293Z Running test: batch_norm_net sync interval 2022-03-17T20:28:19.4142140Z 2 with iteration mapping 2022-03-17T20:28:19.4142791Z {0: 5, 1: 15} 2022-03-17T20:28:19.4188431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.4189728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.4411305Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.4413028Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.4415392Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.4417056Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.4469492Z Running test: large_conv_model sync interval 2022-03-17T20:28:19.4470333Z 3 with iteration mapping 2022-03-17T20:28:19.4470995Z {0: 0, 1: 2} 2022-03-17T20:28:19.6342400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:19.8176655Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.8178364Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.8180748Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:19.8182392Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:19.8237687Z Running test: large_conv_model sync interval 2022-03-17T20:28:19.8238566Z 3 with iteration mapping 2022-03-17T20:28:19.8239229Z {0: 0, 1: 3} 2022-03-17T20:28:20.0085760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:20.3739297Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:20.3741286Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:20.3743685Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:20.3745347Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:20.3791643Z Running test: large_conv_model sync interval 2022-03-17T20:28:20.3792497Z 3 with iteration mapping 2022-03-17T20:28:20.3793172Z {0: 0, 1: 10} 2022-03-17T20:28:20.5641599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:22.2064056Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:22.2065843Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:22.2068216Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:22.2069875Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:22.2119324Z Running test: large_conv_model sync interval 2022-03-17T20:28:22.2120174Z 3 with iteration mapping 2022-03-17T20:28:22.2120852Z {0: 5, 1: 7} 2022-03-17T20:28:22.3967363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:22.3968924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:23.4923795Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:23.4925859Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:23.4928704Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:23.4930655Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:23.4977960Z Running test: large_conv_model sync interval 2022-03-17T20:28:23.4979074Z 3 with iteration mapping 2022-03-17T20:28:23.4979932Z {0: 5, 1: 8} 2022-03-17T20:28:23.6825068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:23.6826761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:24.9939487Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:24.9941840Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:24.9944655Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:24.9946647Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:24.9993154Z Running test: large_conv_model sync interval 2022-03-17T20:28:24.9994061Z 3 with iteration mapping 2022-03-17T20:28:24.9994762Z {0: 5, 1: 15} 2022-03-17T20:28:25.1842847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:25.1844108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7415190Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.7417271Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.7420081Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.7422009Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.7467532Z Running test: small_model sync interval 2022-03-17T20:28:27.7468526Z 4 with iteration mapping 2022-03-17T20:28:27.7469312Z {0: 0, 1: 2} 2022-03-17T20:28:27.7504691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7548643Z Running test: small_model sync interval 2022-03-17T20:28:27.7550104Z 4 with iteration mapping 2022-03-17T20:28:27.7550982Z {0: 0, 1: 3} 2022-03-17T20:28:27.7584653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7637705Z Running test: small_model sync interval 2022-03-17T20:28:27.7638741Z 4 with iteration mapping 2022-03-17T20:28:27.7639570Z {0: 0, 1: 10} 2022-03-17T20:28:27.7671905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7788804Z Running test: small_model sync interval 2022-03-17T20:28:27.7789884Z 4 with iteration mapping 2022-03-17T20:28:27.7790720Z {0: 5, 1: 7} 2022-03-17T20:28:27.7825513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7826816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7915122Z Running test: small_model sync interval 2022-03-17T20:28:27.7916144Z 4 with iteration mapping 2022-03-17T20:28:27.7916992Z {0: 5, 1: 8} 2022-03-17T20:28:27.7951215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.7952843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.8050392Z Running test: small_model sync interval 2022-03-17T20:28:27.8051424Z 4 with iteration mapping 2022-03-17T20:28:27.8052530Z {0: 5, 1: 15} 2022-03-17T20:28:27.8081395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.8083222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.8241124Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8242475Z 5 with iteration mapping 2022-03-17T20:28:27.8243361Z {0: 0, 1: 2} 2022-03-17T20:28:27.8284880Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8287087Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8290036Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8292086Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8303783Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8304787Z 5 with iteration mapping 2022-03-17T20:28:27.8305430Z {0: 0, 1: 3} 2022-03-17T20:28:27.8355200Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8357241Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8360273Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8362630Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8376031Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8376710Z 5 with iteration mapping 2022-03-17T20:28:27.8377201Z {0: 0, 1: 10} 2022-03-17T20:28:27.8472694Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8474163Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8476218Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8477608Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8493321Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8494238Z 5 with iteration mapping 2022-03-17T20:28:27.8495202Z {0: 5, 1: 7} 2022-03-17T20:28:27.8573877Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8576161Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8579053Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8581103Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8596116Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8596897Z 5 with iteration mapping 2022-03-17T20:28:27.8597512Z {0: 5, 1: 8} 2022-03-17T20:28:27.8684416Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8686461Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8689189Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8691139Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8703137Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8703927Z 5 with iteration mapping 2022-03-17T20:28:27.8704526Z {0: 5, 1: 15} 2022-03-17T20:28:27.8840978Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8843208Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8845959Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8847872Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8859357Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8860137Z 6 with iteration mapping 2022-03-17T20:28:27.8860749Z {0: 0, 1: 2} 2022-03-17T20:28:27.8903076Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8904927Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8908014Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8909998Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8923268Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8923983Z 6 with iteration mapping 2022-03-17T20:28:27.8924631Z {0: 0, 1: 3} 2022-03-17T20:28:27.8974744Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8976223Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8978217Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.8979617Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.8992341Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.8993426Z 6 with iteration mapping 2022-03-17T20:28:27.8994165Z {0: 0, 1: 10} 2022-03-17T20:28:27.9098857Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9101088Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9104359Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9106313Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9114169Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.9114976Z 6 with iteration mapping 2022-03-17T20:28:27.9115638Z {0: 5, 1: 7} 2022-03-17T20:28:27.9201396Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9203787Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9206497Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9208456Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9223178Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.9224009Z 6 with iteration mapping 2022-03-17T20:28:27.9224634Z {0: 5, 1: 8} 2022-03-17T20:28:27.9312125Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9314135Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9316896Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9318854Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9333277Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:27.9334120Z 6 with iteration mapping 2022-03-17T20:28:27.9334747Z {0: 5, 1: 15} 2022-03-17T20:28:27.9475068Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9477112Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9479847Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:27.9481768Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:27.9492941Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:27.9494055Z 7 with iteration mapping 2022-03-17T20:28:27.9494698Z {0: 0, 1: 2} 2022-03-17T20:28:27.9528497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.9571979Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:27.9572777Z 7 with iteration mapping 2022-03-17T20:28:27.9573419Z {0: 0, 1: 3} 2022-03-17T20:28:27.9608662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.9659044Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:27.9659866Z 7 with iteration mapping 2022-03-17T20:28:27.9660487Z {0: 0, 1: 10} 2022-03-17T20:28:27.9693788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.9801624Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:27.9802806Z 7 with iteration mapping 2022-03-17T20:28:27.9803447Z {0: 5, 1: 7} 2022-03-17T20:28:27.9836001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.9837256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.9912270Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:27.9913154Z 7 with iteration mapping 2022-03-17T20:28:27.9914172Z {0: 5, 1: 8} 2022-03-17T20:28:27.9949113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:27.9950306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0044063Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:28.0044881Z 7 with iteration mapping 2022-03-17T20:28:28.0045532Z {0: 5, 1: 15} 2022-03-17T20:28:28.0080066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0081255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0226125Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:28.0226951Z 8 with iteration mapping 2022-03-17T20:28:28.0227559Z {0: 0, 1: 2} 2022-03-17T20:28:28.0261564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0306736Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:28.0307585Z 8 with iteration mapping 2022-03-17T20:28:28.0308218Z {0: 0, 1: 3} 2022-03-17T20:28:28.0340561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0395574Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:28.0396375Z 8 with iteration mapping 2022-03-17T20:28:28.0397023Z {0: 0, 1: 10} 2022-03-17T20:28:28.0427440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0534560Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:28.0535409Z 8 with iteration mapping 2022-03-17T20:28:28.0536041Z {0: 5, 1: 7} 2022-03-17T20:28:28.0571545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0572769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0647944Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:28.0648794Z 8 with iteration mapping 2022-03-17T20:28:28.0649433Z {0: 5, 1: 8} 2022-03-17T20:28:28.0682775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0684014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0763760Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:28.0764614Z 8 with iteration mapping 2022-03-17T20:28:28.0765243Z {0: 5, 1: 15} 2022-03-17T20:28:28.0798641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0799989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.0936031Z Running test: batch_norm_net sync interval 2022-03-17T20:28:28.0936800Z 1 with iteration mapping 2022-03-17T20:28:28.0937433Z {0: 0, 1: 2} 2022-03-17T20:28:28.0965484Z Running test: batch_norm_net sync interval 2022-03-17T20:28:28.0966288Z 1 with iteration mapping 2022-03-17T20:28:28.0966929Z {0: 0, 1: 3} 2022-03-17T20:28:28.0986788Z Running test: batch_norm_net sync interval 2022-03-17T20:28:28.0987570Z 1 with iteration mapping 2022-03-17T20:28:28.0988176Z {0: 0, 1: 10} 2022-03-17T20:28:28.1006951Z Running test: batch_norm_net sync interval 2022-03-17T20:28:28.1007732Z 1 with iteration mapping 2022-03-17T20:28:28.1008339Z {0: 5, 1: 7} 2022-03-17T20:28:28.1057669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.1059165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.1153990Z Running test: batch_norm_net sync interval 2022-03-17T20:28:28.1154761Z 1 with iteration mapping 2022-03-17T20:28:28.1155385Z {0: 5, 1: 8} 2022-03-17T20:28:28.1206114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.1207336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.1320915Z Running test: batch_norm_net sync interval 2022-03-17T20:28:28.1321676Z 1 with iteration mapping 2022-03-17T20:28:28.1322474Z {0: 5, 1: 15} 2022-03-17T20:28:28.1368511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.1369940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.1489311Z Running test: large_conv_model sync interval 2022-03-17T20:28:28.1490085Z 1 with iteration mapping 2022-03-17T20:28:28.1490707Z {0: 0, 1: 2} 2022-03-17T20:28:28.1514313Z Running test: large_conv_model sync interval 2022-03-17T20:28:28.1515087Z 1 with iteration mapping 2022-03-17T20:28:28.1515704Z {0: 0, 1: 3} 2022-03-17T20:28:28.1535760Z Running test: large_conv_model sync interval 2022-03-17T20:28:28.1536573Z 1 with iteration mapping 2022-03-17T20:28:28.1537190Z {0: 0, 1: 10} 2022-03-17T20:28:28.1555834Z Running test: large_conv_model sync interval 2022-03-17T20:28:28.1556623Z 1 with iteration mapping 2022-03-17T20:28:28.1557227Z {0: 5, 1: 7} 2022-03-17T20:28:28.3415567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:28.3416820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:29.0743792Z Running test: large_conv_model sync interval 2022-03-17T20:28:29.0744588Z 1 with iteration mapping 2022-03-17T20:28:29.0745189Z {0: 5, 1: 8} 2022-03-17T20:28:29.2605236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:29.2606845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:29.9932006Z Running test: large_conv_model sync interval 2022-03-17T20:28:29.9932776Z 1 with iteration mapping 2022-03-17T20:28:29.9933396Z {0: 5, 1: 15} 2022-03-17T20:28:30.1797793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.1799005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9116345Z Running test: small_model sync interval 2022-03-17T20:28:30.9117139Z 1 with iteration mapping 2022-03-17T20:28:30.9117756Z {0: 0, 1: 2} 2022-03-17T20:28:30.9137294Z Running test: small_model sync interval 2022-03-17T20:28:30.9138066Z 1 with iteration mapping 2022-03-17T20:28:30.9138669Z {0: 0, 1: 3} 2022-03-17T20:28:30.9154581Z Running test: small_model sync interval 2022-03-17T20:28:30.9155359Z 1 with iteration mapping 2022-03-17T20:28:30.9155957Z {0: 0, 1: 10} 2022-03-17T20:28:30.9173517Z Running test: small_model sync interval 2022-03-17T20:28:30.9174268Z 1 with iteration mapping 2022-03-17T20:28:30.9174884Z {0: 5, 1: 7} 2022-03-17T20:28:30.9218439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9220006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9291730Z Running test: small_model sync interval 2022-03-17T20:28:30.9292502Z 1 with iteration mapping 2022-03-17T20:28:30.9293105Z {0: 5, 1: 8} 2022-03-17T20:28:30.9334651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9335879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9422055Z Running test: small_model sync interval 2022-03-17T20:28:30.9422831Z 1 with iteration mapping 2022-03-17T20:28:30.9423433Z {0: 5, 1: 15} 2022-03-17T20:28:30.9475295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9476486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:30.9576213Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:30.9577128Z 1 with iteration mapping 2022-03-17T20:28:30.9577733Z {0: 0, 1: 2} 2022-03-17T20:28:30.9600706Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:30.9601576Z 1 with iteration mapping 2022-03-17T20:28:30.9602357Z {0: 0, 1: 3} 2022-03-17T20:28:30.9626757Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:30.9627616Z 1 with iteration mapping 2022-03-17T20:28:30.9628242Z {0: 0, 1: 10} 2022-03-17T20:28:30.9652846Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:30.9653708Z 1 with iteration mapping 2022-03-17T20:28:30.9654336Z {0: 5, 1: 7} 2022-03-17T20:28:30.9762155Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:30.9763070Z 1 with iteration mapping 2022-03-17T20:28:30.9763684Z {0: 5, 1: 8} 2022-03-17T20:28:30.9896321Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:28:30.9897186Z 1 with iteration mapping 2022-03-17T20:28:30.9897806Z {0: 5, 1: 15} 2022-03-17T20:28:31.0041053Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:31.0042793Z 1 with iteration mapping 2022-03-17T20:28:31.0043452Z {0: 0, 1: 2} 2022-03-17T20:28:31.0070202Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:31.0071063Z 1 with iteration mapping 2022-03-17T20:28:31.0071668Z {0: 0, 1: 3} 2022-03-17T20:28:31.0096986Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:31.0097855Z 1 with iteration mapping 2022-03-17T20:28:31.0098473Z {0: 0, 1: 10} 2022-03-17T20:28:31.0123789Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:31.0124655Z 1 with iteration mapping 2022-03-17T20:28:31.0125264Z {0: 5, 1: 7} 2022-03-17T20:28:31.0223620Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:31.0224520Z 1 with iteration mapping 2022-03-17T20:28:31.0225184Z {0: 5, 1: 8} 2022-03-17T20:28:31.0346942Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:28:31.0347798Z 1 with iteration mapping 2022-03-17T20:28:31.0348421Z {0: 5, 1: 15} 2022-03-17T20:28:31.0459080Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:31.0459937Z 1 with iteration mapping 2022-03-17T20:28:31.0460574Z {0: 0, 1: 2} 2022-03-17T20:28:31.0477555Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:31.0478390Z 1 with iteration mapping 2022-03-17T20:28:31.0478991Z {0: 0, 1: 3} 2022-03-17T20:28:31.0495682Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:31.0496516Z 1 with iteration mapping 2022-03-17T20:28:31.0497136Z {0: 0, 1: 10} 2022-03-17T20:28:31.0518945Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:31.0519767Z 1 with iteration mapping 2022-03-17T20:28:31.0520394Z {0: 5, 1: 7} 2022-03-17T20:28:31.0569319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0570532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0649812Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:31.0650667Z 1 with iteration mapping 2022-03-17T20:28:31.0651300Z {0: 5, 1: 8} 2022-03-17T20:28:31.0686092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0687308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0762826Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:28:31.0763663Z 1 with iteration mapping 2022-03-17T20:28:31.0764310Z {0: 5, 1: 15} 2022-03-17T20:28:31.0809756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0810981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0893195Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:31.0894022Z 1 with iteration mapping 2022-03-17T20:28:31.0894636Z {0: 0, 1: 2} 2022-03-17T20:28:31.0913533Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:31.0914361Z 1 with iteration mapping 2022-03-17T20:28:31.0914967Z {0: 0, 1: 3} 2022-03-17T20:28:31.0927460Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:31.0928277Z 1 with iteration mapping 2022-03-17T20:28:31.0928911Z {0: 0, 1: 10} 2022-03-17T20:28:31.0950680Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:31.0955521Z 1 with iteration mapping 2022-03-17T20:28:31.0956184Z {0: 5, 1: 7} 2022-03-17T20:28:31.0990597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.0991882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.1063496Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:31.1064662Z 1 with iteration mapping 2022-03-17T20:28:31.1065286Z {0: 5, 1: 8} 2022-03-17T20:28:31.1129428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.1130772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.1194163Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:28:31.1195325Z 1 with iteration mapping 2022-03-17T20:28:31.1195946Z {0: 5, 1: 15} 2022-03-17T20:28:31.1243167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:31.1244076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:33.9891472Z ok (72.274s) 2022-03-17T20:28:33.9891989Z 2022-03-17T20:28:33.9893304Z ---------------------------------------------------------------------- 2022-03-17T20:28:33.9894635Z Ran 1 test in 72.275s 2022-03-17T20:28:33.9895011Z 2022-03-17T20:28:33.9895190Z OK 2022-03-17T20:28:33.9895513Z 2022-03-17T20:28:33.9895797Z Generating XML reports... 2022-03-17T20:28:33.9966601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202721.xml 2022-03-17T20:28:36.6305725Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:36.6315427Z 2022-03-17T20:28:36.6316216Z Running tests... 2022-03-17T20:28:36.6317797Z ---------------------------------------------------------------------- 2022-03-17T20:28:37.2381823Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103283 2022-03-17T20:28:37.2428553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103284 2022-03-17T20:28:37.2430215Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:28:38.3560951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:38.3744624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:38.3746277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:38.3764693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:38.3768443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:38.4760771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:38.8588725Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:28:38.9060275Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:28:48.1382410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:48.1383723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:48.1581917Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:48.1584128Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:48.1613712Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:48.1615873Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:48.2393195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:48.2394516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:28:48.2456593Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:28:48.2458758Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:28:50.8768986Z ok (14.245s) 2022-03-17T20:28:50.8770643Z 2022-03-17T20:28:50.8772104Z ---------------------------------------------------------------------- 2022-03-17T20:28:50.8773368Z Ran 1 test in 14.245s 2022-03-17T20:28:50.8773764Z 2022-03-17T20:28:50.8773977Z OK 2022-03-17T20:28:50.8774433Z 2022-03-17T20:28:50.8774873Z Generating XML reports... 2022-03-17T20:28:50.8853401Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202836.xml 2022-03-17T20:28:53.6757179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:53.6772194Z 2022-03-17T20:28:53.6772767Z Running tests... 2022-03-17T20:28:53.6773480Z ---------------------------------------------------------------------- 2022-03-17T20:28:54.3147732Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103502 2022-03-17T20:28:54.3194047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103503 2022-03-17T20:28:54.3195453Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:28:55.4518700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:55.4819403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:55.4822213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:55.4824148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:55.4828599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:55.5829531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:55.8615506Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:28:55.9607049Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:29:07.6531583Z ok (13.975s) 2022-03-17T20:29:07.6532747Z 2022-03-17T20:29:07.6534150Z ---------------------------------------------------------------------- 2022-03-17T20:29:07.6535046Z Ran 1 test in 13.976s 2022-03-17T20:29:07.6535418Z 2022-03-17T20:29:07.6535610Z OK 2022-03-17T20:29:07.6535917Z 2022-03-17T20:29:07.6536957Z Generating XML reports... 2022-03-17T20:29:07.6604550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202853.xml 2022-03-17T20:29:10.3306620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:10.3314737Z 2022-03-17T20:29:10.3315003Z Running tests... 2022-03-17T20:29:10.3318004Z ---------------------------------------------------------------------- 2022-03-17T20:29:10.9391565Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103721 2022-03-17T20:29:10.9423191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103722 2022-03-17T20:29:10.9424605Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:29:12.0282437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:12.0627936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:12.0628912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:12.0685546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:12.0688016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:12.0690129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:29:12.1639937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:12.1641794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:29:12.1645041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:29:12.1707623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:29:14.0512538Z ok (3.719s) 2022-03-17T20:29:14.0513207Z 2022-03-17T20:29:14.0514303Z ---------------------------------------------------------------------- 2022-03-17T20:29:14.0515094Z Ran 1 test in 3.720s 2022-03-17T20:29:14.0515456Z 2022-03-17T20:29:14.0517422Z OK 2022-03-17T20:29:14.0517867Z 2022-03-17T20:29:14.0518167Z Generating XML reports... 2022-03-17T20:29:14.0588213Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202910.xml 2022-03-17T20:29:16.6983187Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:16.6996077Z 2022-03-17T20:29:16.6996560Z Running tests... 2022-03-17T20:29:16.6997741Z ---------------------------------------------------------------------- 2022-03-17T20:29:17.3132282Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103933 2022-03-17T20:29:17.3189351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103934 2022-03-17T20:29:17.3190948Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:29:18.4548352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:18.4684774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:18.4685756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:18.4749092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:18.4753446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:18.4756027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:29:18.5699116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:18.5700831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:29:18.5702837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:29:18.5779965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:29:20.4288408Z ok (3.729s) 2022-03-17T20:29:20.4289178Z 2022-03-17T20:29:20.4290512Z ---------------------------------------------------------------------- 2022-03-17T20:29:20.4291734Z Ran 1 test in 3.729s 2022-03-17T20:29:20.4292120Z 2022-03-17T20:29:20.4292337Z OK 2022-03-17T20:29:20.4292649Z 2022-03-17T20:29:20.4292931Z Generating XML reports... 2022-03-17T20:29:20.4352697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202916.xml 2022-03-17T20:29:23.1449848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:23.1459087Z 2022-03-17T20:29:23.1459720Z Running tests... 2022-03-17T20:29:23.1460839Z ---------------------------------------------------------------------- 2022-03-17T20:29:23.7731056Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104145 2022-03-17T20:29:23.7781031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104146 2022-03-17T20:29:23.7782634Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:29:24.9029952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:24.9106790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:24.9108773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:24.9130620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:24.9137133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:25.0120175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:25.3948231Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:29:25.4624453Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:29:34.7125969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:34.7127918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:34.7202593Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:29:34.7207574Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:29:37.4073099Z ok (14.261s) 2022-03-17T20:29:37.4074300Z 2022-03-17T20:29:37.4075398Z ---------------------------------------------------------------------- 2022-03-17T20:29:37.4076239Z Ran 1 test in 14.261s 2022-03-17T20:29:37.4076613Z 2022-03-17T20:29:37.4076822Z OK 2022-03-17T20:29:37.4077106Z 2022-03-17T20:29:37.4077391Z Generating XML reports... 2022-03-17T20:29:37.4146428Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202923.xml 2022-03-17T20:29:40.0870596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:40.0882557Z 2022-03-17T20:29:40.0883517Z Running tests... 2022-03-17T20:29:40.0884948Z ---------------------------------------------------------------------- 2022-03-17T20:29:40.7155673Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104364 2022-03-17T20:29:40.7199945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104365 2022-03-17T20:29:40.7201333Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:29:41.8208344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:41.8737171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:41.8739313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:41.8812216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:41.8816049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:41.9749541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:42.3660901Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:29:42.4139010Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:29:51.4965792Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:29:51.6736611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:51.6737828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:54.3477265Z ok (14.259s) 2022-03-17T20:29:54.3477645Z 2022-03-17T20:29:54.3479287Z ---------------------------------------------------------------------- 2022-03-17T20:29:54.3480333Z Ran 1 test in 14.259s 2022-03-17T20:29:54.3480805Z 2022-03-17T20:29:54.3481026Z OK 2022-03-17T20:29:54.3481318Z 2022-03-17T20:29:54.3481592Z Generating XML reports... 2022-03-17T20:29:54.3551345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202940.xml 2022-03-17T20:29:56.9950393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:56.9961172Z 2022-03-17T20:29:56.9962785Z Running tests... 2022-03-17T20:29:56.9964256Z ---------------------------------------------------------------------- 2022-03-17T20:29:57.6005394Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104583 2022-03-17T20:29:57.6044993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104584 2022-03-17T20:29:57.6046632Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:29:58.7276647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:58.7412399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:58.7414701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:58.7477346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:58.7480898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:58.8428570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:00.5142642Z ok (3.518s) 2022-03-17T20:30:00.5143178Z 2022-03-17T20:30:00.5144355Z ---------------------------------------------------------------------- 2022-03-17T20:30:00.5146070Z Ran 1 test in 3.518s 2022-03-17T20:30:00.5146519Z 2022-03-17T20:30:00.5146764Z OK 2022-03-17T20:30:00.5152299Z 2022-03-17T20:30:00.5152940Z Generating XML reports... 2022-03-17T20:30:00.5223002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202956.xml 2022-03-17T20:30:03.1466055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:03.1478142Z 2022-03-17T20:30:03.1478294Z Running tests... 2022-03-17T20:30:03.1478889Z ---------------------------------------------------------------------- 2022-03-17T20:30:03.1497357Z test_gather (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:30:03.1498030Z 2022-03-17T20:30:03.1498963Z ---------------------------------------------------------------------- 2022-03-17T20:30:03.1499759Z Ran 1 test in 0.002s 2022-03-17T20:30:03.1500137Z 2022-03-17T20:30:03.1500400Z OK (skipped=1) 2022-03-17T20:30:03.1500751Z 2022-03-17T20:30:03.1501028Z Generating XML reports... 2022-03-17T20:30:03.1534006Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203003.xml 2022-03-17T20:30:05.8481660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:05.8492178Z 2022-03-17T20:30:05.8492865Z Running tests... 2022-03-17T20:30:05.8494085Z ---------------------------------------------------------------------- 2022-03-17T20:30:05.8516743Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:05.8517406Z 2022-03-17T20:30:05.8518206Z ---------------------------------------------------------------------- 2022-03-17T20:30:05.8518978Z Ran 1 test in 0.002s 2022-03-17T20:30:05.8519333Z 2022-03-17T20:30:05.8519565Z OK (skipped=1) 2022-03-17T20:30:05.8519909Z 2022-03-17T20:30:05.8520176Z Generating XML reports... 2022-03-17T20:30:05.8558128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203005.xml 2022-03-17T20:30:08.8949239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:08.8959558Z 2022-03-17T20:30:08.8960281Z Running tests... 2022-03-17T20:30:08.8961491Z ---------------------------------------------------------------------- 2022-03-17T20:30:09.5003138Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104925 2022-03-17T20:30:09.5064589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104926 2022-03-17T20:30:09.5065966Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:10.6288773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:10.6417786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:10.6419875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:10.6492284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:10.6501758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:10.7428223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:11.0372283Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:11.1199887Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:13.0161431Z ok (4.120s) 2022-03-17T20:30:13.0162081Z 2022-03-17T20:30:13.0162988Z ---------------------------------------------------------------------- 2022-03-17T20:30:13.0164327Z Ran 1 test in 4.120s 2022-03-17T20:30:13.0164691Z 2022-03-17T20:30:13.0164875Z OK 2022-03-17T20:30:13.0165172Z 2022-03-17T20:30:13.0165449Z Generating XML reports... 2022-03-17T20:30:13.0243685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203008.xml 2022-03-17T20:30:15.6438049Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:15.6445999Z 2022-03-17T20:30:15.6446491Z Running tests... 2022-03-17T20:30:15.6447625Z ---------------------------------------------------------------------- 2022-03-17T20:30:15.6463038Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:30:15.6463639Z 2022-03-17T20:30:15.6464403Z ---------------------------------------------------------------------- 2022-03-17T20:30:15.6465193Z Ran 1 test in 0.002s 2022-03-17T20:30:15.6465560Z 2022-03-17T20:30:15.6465809Z OK (skipped=1) 2022-03-17T20:30:15.6466172Z 2022-03-17T20:30:15.6466450Z Generating XML reports... 2022-03-17T20:30:15.6500177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203015.xml 2022-03-17T20:30:18.3695796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:18.3707416Z 2022-03-17T20:30:18.3707932Z Running tests... 2022-03-17T20:30:18.3709109Z ---------------------------------------------------------------------- 2022-03-17T20:30:18.3723468Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:30:18.3723752Z 2022-03-17T20:30:18.3724431Z ---------------------------------------------------------------------- 2022-03-17T20:30:18.3724771Z Ran 1 test in 0.002s 2022-03-17T20:30:18.3724949Z 2022-03-17T20:30:18.3725057Z OK (skipped=1) 2022-03-17T20:30:18.3725224Z 2022-03-17T20:30:18.3725338Z Generating XML reports... 2022-03-17T20:30:18.3762211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203018.xml 2022-03-17T20:30:21.1542192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:21.1553992Z 2022-03-17T20:30:21.1554430Z Running tests... 2022-03-17T20:30:21.1556212Z ---------------------------------------------------------------------- 2022-03-17T20:30:21.7573901Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105280 2022-03-17T20:30:21.7633143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105281 2022-03-17T20:30:21.7634549Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:22.9029676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:22.9034320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:22.9035224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:22.9135051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:22.9136404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:23.0052616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:23.7124540Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:23.7724800Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:25.5739764Z ok (4.418s) 2022-03-17T20:30:25.5740362Z 2022-03-17T20:30:25.5741495Z ---------------------------------------------------------------------- 2022-03-17T20:30:25.5742909Z Ran 1 test in 4.418s 2022-03-17T20:30:25.5743272Z 2022-03-17T20:30:25.5743465Z OK 2022-03-17T20:30:25.5743752Z 2022-03-17T20:30:25.5744037Z Generating XML reports... 2022-03-17T20:30:25.5821574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203021.xml 2022-03-17T20:30:28.2478237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:28.2487631Z 2022-03-17T20:30:28.2488231Z Running tests... 2022-03-17T20:30:28.2489720Z ---------------------------------------------------------------------- 2022-03-17T20:30:28.8513159Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105495 2022-03-17T20:30:28.8546065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105496 2022-03-17T20:30:28.8547679Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:29.9367568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:29.9645493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:29.9647640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:29.9676432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:29.9685405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:30.0652785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:30.0813662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:30:30.0815882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:30:30.0817194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:30:30.0818840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:30:30.8247867Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:30.8855329Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:31.1258141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:30:31.1260125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:30:31.1263738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:30:31.1360641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:30:31.1643570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:30:31.1644813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:30:31.1646720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:30:31.1748288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:30:32.8650350Z ok (4.615s) 2022-03-17T20:30:32.8650903Z 2022-03-17T20:30:32.8652156Z ---------------------------------------------------------------------- 2022-03-17T20:30:32.8653064Z Ran 1 test in 4.616s 2022-03-17T20:30:32.8653440Z 2022-03-17T20:30:32.8653614Z OK 2022-03-17T20:30:32.8653915Z 2022-03-17T20:30:32.8654942Z Generating XML reports... 2022-03-17T20:30:32.8730293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203028.xml 2022-03-17T20:30:35.8578299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:35.8589270Z 2022-03-17T20:30:35.8589416Z Running tests... 2022-03-17T20:30:35.8590840Z ---------------------------------------------------------------------- 2022-03-17T20:30:36.4992820Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105749 2022-03-17T20:30:36.5039713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105750 2022-03-17T20:30:36.5041016Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:37.5799419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:37.5800736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:37.5802947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:37.5899920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:37.5901326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:37.5902682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:30:37.6812524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:37.6813809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:30:37.6815671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:30:37.6817562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:30:39.4123739Z ok (3.553s) 2022-03-17T20:30:39.4124113Z 2022-03-17T20:30:39.4125202Z ---------------------------------------------------------------------- 2022-03-17T20:30:39.4126022Z Ran 1 test in 3.553s 2022-03-17T20:30:39.4126395Z 2022-03-17T20:30:39.4127090Z OK 2022-03-17T20:30:39.4127413Z 2022-03-17T20:30:39.4127693Z Generating XML reports... 2022-03-17T20:30:39.4209519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203035.xml 2022-03-17T20:30:42.0187488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:42.0198733Z 2022-03-17T20:30:42.0199526Z Running tests... 2022-03-17T20:30:42.0200673Z ---------------------------------------------------------------------- 2022-03-17T20:30:42.6612703Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105961 2022-03-17T20:30:42.6662181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105962 2022-03-17T20:30:42.6663461Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:43.7744903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:43.7987107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:43.7989087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:43.8047607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:43.8049663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:43.9000956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:44.1835288Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:44.2830700Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:30:46.1764561Z ok (4.156s) 2022-03-17T20:30:46.1764972Z 2022-03-17T20:30:46.1765857Z ---------------------------------------------------------------------- 2022-03-17T20:30:46.1766639Z Ran 1 test in 4.157s 2022-03-17T20:30:46.1767006Z 2022-03-17T20:30:46.1767201Z OK 2022-03-17T20:30:46.1767500Z 2022-03-17T20:30:46.1767760Z Generating XML reports... 2022-03-17T20:30:46.1853668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203042.xml 2022-03-17T20:30:48.7825995Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:48.7837448Z 2022-03-17T20:30:48.7838042Z Running tests... 2022-03-17T20:30:48.7839191Z ---------------------------------------------------------------------- 2022-03-17T20:30:49.3868535Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106176 2022-03-17T20:30:49.3903257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106177 2022-03-17T20:30:49.3904631Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:50.5267544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:50.5628537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:50.5630426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:50.5672449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:50.5675002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:50.6645765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:52.4992056Z ok (3.715s) 2022-03-17T20:30:52.4992728Z 2022-03-17T20:30:52.4993723Z ---------------------------------------------------------------------- 2022-03-17T20:30:52.4994476Z Ran 1 test in 3.715s 2022-03-17T20:30:52.4994835Z 2022-03-17T20:30:52.4995016Z OK 2022-03-17T20:30:52.4995301Z 2022-03-17T20:30:52.4995571Z Generating XML reports... 2022-03-17T20:30:52.5069728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203048.xml 2022-03-17T20:30:55.1359196Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:55.1371916Z 2022-03-17T20:30:55.1372318Z Running tests... 2022-03-17T20:30:55.1373911Z ---------------------------------------------------------------------- 2022-03-17T20:30:55.7426294Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106384 2022-03-17T20:30:55.7461800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106385 2022-03-17T20:30:55.7463109Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:30:56.8157901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:56.8512676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:56.8514584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:56.8565791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:56.8573468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:56.8576915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:30:56.9520574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:56.9522600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:30:56.9524437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:30:56.9599917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:30:58.7555823Z ok (3.618s) 2022-03-17T20:30:58.7556225Z 2022-03-17T20:30:58.7557043Z ---------------------------------------------------------------------- 2022-03-17T20:30:58.7557823Z Ran 1 test in 3.619s 2022-03-17T20:30:58.7558209Z 2022-03-17T20:30:58.7558383Z OK 2022-03-17T20:30:58.7558673Z 2022-03-17T20:30:58.7558946Z Generating XML reports... 2022-03-17T20:30:58.7638531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203055.xml 2022-03-17T20:31:01.3467687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:01.3480316Z 2022-03-17T20:31:01.3480567Z Running tests... 2022-03-17T20:31:01.3481099Z ---------------------------------------------------------------------- 2022-03-17T20:31:01.9519942Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106596 2022-03-17T20:31:01.9561332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106597 2022-03-17T20:31:01.9562883Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:31:03.0233278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:03.0264833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:03.0266898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:03.0334309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:03.0339403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:03.0343343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:31:03.1275295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:03.1277436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:31:03.1279833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:31:03.1366543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:31:04.9659135Z ok (3.618s) 2022-03-17T20:31:04.9659681Z 2022-03-17T20:31:04.9661194Z ---------------------------------------------------------------------- 2022-03-17T20:31:04.9662210Z Ran 1 test in 3.618s 2022-03-17T20:31:04.9662627Z 2022-03-17T20:31:04.9662811Z OK 2022-03-17T20:31:04.9663104Z 2022-03-17T20:31:04.9663382Z Generating XML reports... 2022-03-17T20:31:04.9741269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203101.xml 2022-03-17T20:31:07.5728957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:07.5741015Z 2022-03-17T20:31:07.5741295Z Running tests... 2022-03-17T20:31:07.5741836Z ---------------------------------------------------------------------- 2022-03-17T20:31:08.2117789Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106808 2022-03-17T20:31:08.2156787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106809 2022-03-17T20:31:08.2158374Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:31:09.2896039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:09.3031726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:09.3032762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:09.3101606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:09.3102920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:09.4043501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:09.6940157Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:31:09.7834482Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:31:21.9447616Z ok (14.370s) 2022-03-17T20:31:21.9448168Z 2022-03-17T20:31:21.9449365Z ---------------------------------------------------------------------- 2022-03-17T20:31:21.9450295Z Ran 1 test in 14.370s 2022-03-17T20:31:21.9450698Z 2022-03-17T20:31:21.9453056Z OK 2022-03-17T20:31:21.9453460Z 2022-03-17T20:31:21.9453767Z Generating XML reports... 2022-03-17T20:31:21.9530843Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203107.xml 2022-03-17T20:31:24.7091483Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:24.7102792Z 2022-03-17T20:31:24.7104053Z Running tests... 2022-03-17T20:31:24.7105257Z ---------------------------------------------------------------------- 2022-03-17T20:31:24.7126783Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:24.7127595Z 2022-03-17T20:31:24.7128611Z ---------------------------------------------------------------------- 2022-03-17T20:31:24.7129561Z Ran 1 test in 0.002s 2022-03-17T20:31:24.7129937Z 2022-03-17T20:31:24.7130163Z OK (skipped=1) 2022-03-17T20:31:24.7130532Z 2022-03-17T20:31:24.7130805Z Generating XML reports... 2022-03-17T20:31:24.7167750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203124.xml 2022-03-17T20:31:27.3803277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:27.3817222Z 2022-03-17T20:31:27.3817932Z Running tests... 2022-03-17T20:31:27.3819138Z ---------------------------------------------------------------------- 2022-03-17T20:31:27.3833401Z test_isend (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:27.3833987Z 2022-03-17T20:31:27.3835083Z ---------------------------------------------------------------------- 2022-03-17T20:31:27.3836068Z Ran 1 test in 0.002s 2022-03-17T20:31:27.3836505Z 2022-03-17T20:31:27.3836736Z OK (skipped=1) 2022-03-17T20:31:27.3837098Z 2022-03-17T20:31:27.3837369Z Generating XML reports... 2022-03-17T20:31:27.3876834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203127.xml 2022-03-17T20:31:30.0447778Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:30.0461229Z 2022-03-17T20:31:30.0462039Z Running tests... 2022-03-17T20:31:30.0463404Z ---------------------------------------------------------------------- 2022-03-17T20:31:30.0477847Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:30.0478742Z 2022-03-17T20:31:30.0479790Z ---------------------------------------------------------------------- 2022-03-17T20:31:30.0480613Z Ran 1 test in 0.002s 2022-03-17T20:31:30.0480980Z 2022-03-17T20:31:30.0481212Z OK (skipped=1) 2022-03-17T20:31:30.0481559Z 2022-03-17T20:31:30.0481818Z Generating XML reports... 2022-03-17T20:31:30.0519867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203130.xml 2022-03-17T20:31:32.8180007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:32.8197308Z 2022-03-17T20:31:32.8197511Z Running tests... 2022-03-17T20:31:32.8198042Z ---------------------------------------------------------------------- 2022-03-17T20:31:32.8214260Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:32.8214571Z 2022-03-17T20:31:32.8215053Z ---------------------------------------------------------------------- 2022-03-17T20:31:32.8215631Z Ran 1 test in 0.002s 2022-03-17T20:31:32.8215795Z 2022-03-17T20:31:32.8215904Z OK (skipped=1) 2022-03-17T20:31:32.8216510Z 2022-03-17T20:31:32.8216752Z Generating XML reports... 2022-03-17T20:31:32.8252325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203132.xml 2022-03-17T20:31:36.1980925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:36.1992798Z 2022-03-17T20:31:36.1993439Z Running tests... 2022-03-17T20:31:36.1994672Z ---------------------------------------------------------------------- 2022-03-17T20:31:36.8146591Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107295 2022-03-17T20:31:36.8198703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107296 2022-03-17T20:31:36.8200675Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:31:37.9426652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:37.9473428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:37.9475633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:37.9528053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:37.9531288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:38.0491439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:39.7291042Z skip (3.529s) 2022-03-17T20:31:39.7291734Z 2022-03-17T20:31:39.7293023Z ---------------------------------------------------------------------- 2022-03-17T20:31:39.7294043Z Ran 1 test in 3.530s 2022-03-17T20:31:39.7294484Z 2022-03-17T20:31:39.7294743Z OK (skipped=1) 2022-03-17T20:31:39.7295084Z 2022-03-17T20:31:39.7295354Z Generating XML reports... 2022-03-17T20:31:39.7382542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203136.xml 2022-03-17T20:31:42.3967218Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:42.3979934Z 2022-03-17T20:31:42.3980288Z Running tests... 2022-03-17T20:31:42.3980715Z ---------------------------------------------------------------------- 2022-03-17T20:31:42.9999395Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107503 2022-03-17T20:31:43.0033790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107504 2022-03-17T20:31:43.0035177Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:31:44.0641395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:44.1239429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:44.1240342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:44.1241297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:44.1246111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:44.2253818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:45.9119683Z skip (3.513s) 2022-03-17T20:31:45.9120736Z 2022-03-17T20:31:45.9122440Z ---------------------------------------------------------------------- 2022-03-17T20:31:45.9123295Z Ran 1 test in 3.514s 2022-03-17T20:31:45.9123711Z 2022-03-17T20:31:45.9123959Z OK (skipped=1) 2022-03-17T20:31:45.9124319Z 2022-03-17T20:31:45.9124593Z Generating XML reports... 2022-03-17T20:31:45.9202792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203142.xml 2022-03-17T20:31:48.5375212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:48.5385958Z 2022-03-17T20:31:48.5386307Z Running tests... 2022-03-17T20:31:48.5386734Z ---------------------------------------------------------------------- 2022-03-17T20:31:48.5408133Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:48.5408583Z 2022-03-17T20:31:48.5409138Z ---------------------------------------------------------------------- 2022-03-17T20:31:48.5409875Z Ran 1 test in 0.002s 2022-03-17T20:31:48.5410104Z 2022-03-17T20:31:48.5410223Z OK (skipped=1) 2022-03-17T20:31:48.5410384Z 2022-03-17T20:31:48.5410504Z Generating XML reports... 2022-03-17T20:31:48.5448932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203148.xml 2022-03-17T20:31:51.2153197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:51.2163312Z 2022-03-17T20:31:51.2163790Z Running tests... 2022-03-17T20:31:51.2164873Z ---------------------------------------------------------------------- 2022-03-17T20:31:51.2191148Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:31:51.2191990Z 2022-03-17T20:31:51.2192923Z ---------------------------------------------------------------------- 2022-03-17T20:31:51.2193702Z Ran 1 test in 0.003s 2022-03-17T20:31:51.2194069Z 2022-03-17T20:31:51.2194296Z OK (skipped=1) 2022-03-17T20:31:51.2194662Z 2022-03-17T20:31:51.2194936Z Generating XML reports... 2022-03-17T20:31:51.2232137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203151.xml 2022-03-17T20:31:54.0125704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:54.0138344Z 2022-03-17T20:31:54.0138898Z Running tests... 2022-03-17T20:31:54.0140041Z ---------------------------------------------------------------------- 2022-03-17T20:31:54.0156101Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:54.0156825Z 2022-03-17T20:31:54.0157817Z ---------------------------------------------------------------------- 2022-03-17T20:31:54.0158792Z Ran 1 test in 0.002s 2022-03-17T20:31:54.0159266Z 2022-03-17T20:31:54.0159526Z OK (skipped=1) 2022-03-17T20:31:54.0159888Z 2022-03-17T20:31:54.0164429Z Generating XML reports... 2022-03-17T20:31:54.0198322Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203154.xml 2022-03-17T20:31:56.7179554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:56.7188867Z 2022-03-17T20:31:56.7189025Z Running tests... 2022-03-17T20:31:56.7189727Z ---------------------------------------------------------------------- 2022-03-17T20:31:56.7209919Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:56.7211016Z 2022-03-17T20:31:56.7212093Z ---------------------------------------------------------------------- 2022-03-17T20:31:56.7212910Z Ran 1 test in 0.002s 2022-03-17T20:31:56.7213294Z 2022-03-17T20:31:56.7213518Z OK (skipped=1) 2022-03-17T20:31:56.7213866Z 2022-03-17T20:31:56.7214147Z Generating XML reports... 2022-03-17T20:31:56.7252052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203156.xml 2022-03-17T20:31:59.4754879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:59.4770867Z 2022-03-17T20:31:59.4771387Z Running tests... 2022-03-17T20:31:59.4772519Z ---------------------------------------------------------------------- 2022-03-17T20:31:59.4791648Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:59.4792340Z 2022-03-17T20:31:59.4793305Z ---------------------------------------------------------------------- 2022-03-17T20:31:59.4794441Z Ran 1 test in 0.002s 2022-03-17T20:31:59.4794809Z 2022-03-17T20:31:59.4795036Z OK (skipped=1) 2022-03-17T20:31:59.4795391Z 2022-03-17T20:31:59.4795674Z Generating XML reports... 2022-03-17T20:31:59.4836318Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203159.xml 2022-03-17T20:32:02.2312168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:02.2323821Z 2022-03-17T20:32:02.2324317Z Running tests... 2022-03-17T20:32:02.2325495Z ---------------------------------------------------------------------- 2022-03-17T20:32:02.8729726Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108046 2022-03-17T20:32:02.8782569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108047 2022-03-17T20:32:02.8783930Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:03.9790807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:03.9988634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:03.9990532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:03.9992700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:03.9998183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:04.0998541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:04.4774964Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:04.5318513Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:06.2894098Z ok (4.056s) 2022-03-17T20:32:06.2894534Z 2022-03-17T20:32:06.2895534Z ---------------------------------------------------------------------- 2022-03-17T20:32:06.2896309Z Ran 1 test in 4.057s 2022-03-17T20:32:06.2896686Z 2022-03-17T20:32:06.2896881Z OK 2022-03-17T20:32:06.2897207Z 2022-03-17T20:32:06.2897486Z Generating XML reports... 2022-03-17T20:32:06.2966277Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203202.xml 2022-03-17T20:32:08.9030994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:08.9043897Z 2022-03-17T20:32:08.9044207Z Running tests... 2022-03-17T20:32:08.9044873Z ---------------------------------------------------------------------- 2022-03-17T20:32:09.5473595Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108261 2022-03-17T20:32:09.5518399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108262 2022-03-17T20:32:09.5519697Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:10.6584546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:10.6901317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:10.6903170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:10.6992607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:10.6999730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:10.7914649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:11.1719794Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:11.2302961Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:12.9617531Z ok (4.057s) 2022-03-17T20:32:12.9617915Z 2022-03-17T20:32:12.9619917Z ---------------------------------------------------------------------- 2022-03-17T20:32:12.9620873Z Ran 1 test in 4.057s 2022-03-17T20:32:12.9621249Z 2022-03-17T20:32:12.9621426Z OK 2022-03-17T20:32:12.9621751Z 2022-03-17T20:32:12.9622032Z Generating XML reports... 2022-03-17T20:32:12.9700255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203208.xml 2022-03-17T20:32:15.6146260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:15.6157801Z 2022-03-17T20:32:15.6158413Z Running tests... 2022-03-17T20:32:15.6159631Z ---------------------------------------------------------------------- 2022-03-17T20:32:16.2220033Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108476 2022-03-17T20:32:16.2260798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108477 2022-03-17T20:32:16.2262172Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:17.3336330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:17.3683913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:17.3685850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:17.3743800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:17.3748899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:17.4690219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:17.7572391Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:17.8462087Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:19.8362702Z ok (4.220s) 2022-03-17T20:32:19.8363082Z 2022-03-17T20:32:19.8363995Z ---------------------------------------------------------------------- 2022-03-17T20:32:19.8364779Z Ran 1 test in 4.220s 2022-03-17T20:32:19.8365136Z 2022-03-17T20:32:19.8365318Z OK 2022-03-17T20:32:19.8365589Z 2022-03-17T20:32:19.8365859Z Generating XML reports... 2022-03-17T20:32:19.8444638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203215.xml 2022-03-17T20:32:22.4542454Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:22.4554676Z 2022-03-17T20:32:22.4555021Z Running tests... 2022-03-17T20:32:22.4555585Z ---------------------------------------------------------------------- 2022-03-17T20:32:23.0605013Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108691 2022-03-17T20:32:23.0644858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108692 2022-03-17T20:32:23.0646165Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:24.1426724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:24.1837190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:24.1839094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:24.1936485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:24.1942434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:24.2845689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:24.6615439Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:24.7260541Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:26.8749078Z ok (4.419s) 2022-03-17T20:32:26.8749596Z 2022-03-17T20:32:26.8750802Z ---------------------------------------------------------------------- 2022-03-17T20:32:26.8751615Z Ran 1 test in 4.419s 2022-03-17T20:32:26.8752119Z 2022-03-17T20:32:26.8752391Z OK 2022-03-17T20:32:26.8752735Z 2022-03-17T20:32:26.8758371Z Generating XML reports... 2022-03-17T20:32:26.8822650Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203222.xml 2022-03-17T20:32:29.5074320Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:29.5086705Z 2022-03-17T20:32:29.5086954Z Running tests... 2022-03-17T20:32:29.5087479Z ---------------------------------------------------------------------- 2022-03-17T20:32:30.1431809Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108906 2022-03-17T20:32:30.1476971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108907 2022-03-17T20:32:30.1478403Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:31.2682470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:31.3051964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:31.3053849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:31.3087572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:31.3092337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:31.4067679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:31.7963763Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:31.8593932Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:32:33.7575305Z ok (4.248s) 2022-03-17T20:32:33.7575695Z 2022-03-17T20:32:33.7576629Z ---------------------------------------------------------------------- 2022-03-17T20:32:33.7577402Z Ran 1 test in 4.249s 2022-03-17T20:32:33.7577764Z 2022-03-17T20:32:33.7577969Z OK 2022-03-17T20:32:33.7578246Z 2022-03-17T20:32:33.7578522Z Generating XML reports... 2022-03-17T20:32:33.7663273Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203229.xml 2022-03-17T20:32:36.3940657Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:36.3952858Z 2022-03-17T20:32:36.3953327Z Running tests... 2022-03-17T20:32:36.3954499Z ---------------------------------------------------------------------- 2022-03-17T20:32:36.3974612Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:32:36.3975246Z 2022-03-17T20:32:36.3976035Z ---------------------------------------------------------------------- 2022-03-17T20:32:36.3976798Z Ran 1 test in 0.002s 2022-03-17T20:32:36.3977163Z 2022-03-17T20:32:36.3977393Z OK (skipped=1) 2022-03-17T20:32:36.3977724Z 2022-03-17T20:32:36.3977990Z Generating XML reports... 2022-03-17T20:32:36.4015488Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203236.xml 2022-03-17T20:32:39.5875487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:39.5888213Z 2022-03-17T20:32:39.5888526Z Running tests... 2022-03-17T20:32:39.5888989Z ---------------------------------------------------------------------- 2022-03-17T20:32:39.5912102Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:32:39.5912380Z 2022-03-17T20:32:39.5912687Z ---------------------------------------------------------------------- 2022-03-17T20:32:39.5912990Z Ran 1 test in 0.003s 2022-03-17T20:32:39.5913149Z 2022-03-17T20:32:39.5913238Z OK (skipped=1) 2022-03-17T20:32:39.5913375Z 2022-03-17T20:32:39.5913482Z Generating XML reports... 2022-03-17T20:32:39.5953044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203239.xml 2022-03-17T20:32:42.2764532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:42.2779123Z 2022-03-17T20:32:42.2779527Z Running tests... 2022-03-17T20:32:42.2780314Z ---------------------------------------------------------------------- 2022-03-17T20:32:42.2800159Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:32:42.2800664Z 2022-03-17T20:32:42.2801109Z ---------------------------------------------------------------------- 2022-03-17T20:32:42.2801551Z Ran 1 test in 0.002s 2022-03-17T20:32:42.2801764Z 2022-03-17T20:32:42.2801960Z OK (skipped=1) 2022-03-17T20:32:42.2802453Z 2022-03-17T20:32:42.2802581Z Generating XML reports... 2022-03-17T20:32:42.2839387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203242.xml 2022-03-17T20:32:44.7455545Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:44.7466368Z 2022-03-17T20:32:44.7466779Z Running tests... 2022-03-17T20:32:44.7468294Z ---------------------------------------------------------------------- 2022-03-17T20:32:45.3856201Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109335 2022-03-17T20:32:45.3903327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109336 2022-03-17T20:32:45.3904917Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:46.4911849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:46.5486946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:46.5488041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:46.5517388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:46.5519819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:46.6504083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:48.2981645Z ok (3.551s) 2022-03-17T20:32:48.2982152Z 2022-03-17T20:32:48.2983201Z ---------------------------------------------------------------------- 2022-03-17T20:32:48.2984170Z Ran 1 test in 3.551s 2022-03-17T20:32:48.2984645Z 2022-03-17T20:32:48.2984872Z OK 2022-03-17T20:32:48.2985174Z 2022-03-17T20:32:48.2985449Z Generating XML reports... 2022-03-17T20:32:48.3052349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203244.xml 2022-03-17T20:32:50.9639058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:50.9649576Z 2022-03-17T20:32:50.9649805Z Running tests... 2022-03-17T20:32:50.9650706Z ---------------------------------------------------------------------- 2022-03-17T20:32:51.5698633Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109543 2022-03-17T20:32:51.5739219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109544 2022-03-17T20:32:51.5740550Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:32:52.6343194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:52.6947189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:52.6949393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:52.6951188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:52.6955595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:52.7961481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:54.4832637Z ok (3.518s) 2022-03-17T20:32:54.4833233Z 2022-03-17T20:32:54.4834230Z ---------------------------------------------------------------------- 2022-03-17T20:32:54.4835669Z Ran 1 test in 3.518s 2022-03-17T20:32:54.4836052Z 2022-03-17T20:32:54.4836244Z OK 2022-03-17T20:32:54.4836540Z 2022-03-17T20:32:54.4836841Z Generating XML reports... 2022-03-17T20:32:54.4909912Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203250.xml 2022-03-17T20:32:57.1791649Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:57.1802060Z 2022-03-17T20:32:57.1802366Z Running tests... 2022-03-17T20:32:57.1802926Z ---------------------------------------------------------------------- 2022-03-17T20:32:57.1819015Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:32:57.1819541Z 2022-03-17T20:32:57.1819911Z ---------------------------------------------------------------------- 2022-03-17T20:32:57.1820260Z Ran 1 test in 0.002s 2022-03-17T20:32:57.1820428Z 2022-03-17T20:32:57.1820533Z OK (skipped=1) 2022-03-17T20:32:57.1820691Z 2022-03-17T20:32:57.1820815Z Generating XML reports... 2022-03-17T20:32:57.1861791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203257.xml 2022-03-17T20:32:59.8903254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:59.8914502Z 2022-03-17T20:32:59.8915136Z Running tests... 2022-03-17T20:32:59.8916472Z ---------------------------------------------------------------------- 2022-03-17T20:32:59.8930381Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:32:59.8931507Z 2022-03-17T20:32:59.8932404Z ---------------------------------------------------------------------- 2022-03-17T20:32:59.8933178Z Ran 1 test in 0.002s 2022-03-17T20:32:59.8933550Z 2022-03-17T20:32:59.8933760Z OK (skipped=1) 2022-03-17T20:32:59.8934110Z 2022-03-17T20:32:59.8934379Z Generating XML reports... 2022-03-17T20:32:59.8971767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203259.xml 2022-03-17T20:33:02.9721718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:02.9729814Z 2022-03-17T20:33:02.9730086Z Running tests... 2022-03-17T20:33:02.9731195Z ---------------------------------------------------------------------- 2022-03-17T20:33:03.5757309Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109885 2022-03-17T20:33:03.5794493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109886 2022-03-17T20:33:03.5795817Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:33:04.6448917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:04.6881694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:04.6882681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:04.6955596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:04.6957884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:04.7890544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:05.0743992Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:05.1763008Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:17.2123217Z ok (14.239s) 2022-03-17T20:33:17.2123624Z 2022-03-17T20:33:17.2124650Z ---------------------------------------------------------------------- 2022-03-17T20:33:17.2126249Z Ran 1 test in 14.239s 2022-03-17T20:33:17.2126826Z 2022-03-17T20:33:17.2127015Z OK 2022-03-17T20:33:17.2127313Z 2022-03-17T20:33:17.2127622Z Generating XML reports... 2022-03-17T20:33:17.2204520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203302.xml 2022-03-17T20:33:20.0077430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:20.0092301Z 2022-03-17T20:33:20.0092538Z Running tests... 2022-03-17T20:33:20.0093054Z ---------------------------------------------------------------------- 2022-03-17T20:33:20.6146186Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110104 2022-03-17T20:33:20.6189245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110105 2022-03-17T20:33:20.6190780Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:33:21.7639564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:21.7861488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:21.7863601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:21.7941234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:21.7945106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:21.8870871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:22.2668892Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:22.3151521Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:34.3471909Z ok (14.337s) 2022-03-17T20:33:34.3472410Z 2022-03-17T20:33:34.3473391Z ---------------------------------------------------------------------- 2022-03-17T20:33:34.3474174Z Ran 1 test in 14.338s 2022-03-17T20:33:34.3474531Z 2022-03-17T20:33:34.3474710Z OK 2022-03-17T20:33:34.3474994Z 2022-03-17T20:33:34.3475252Z Generating XML reports... 2022-03-17T20:33:34.3548097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203320.xml 2022-03-17T20:33:36.9628407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:36.9640924Z 2022-03-17T20:33:36.9641111Z Running tests... 2022-03-17T20:33:36.9641659Z ---------------------------------------------------------------------- 2022-03-17T20:33:37.5721391Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110323 2022-03-17T20:33:37.5765036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110324 2022-03-17T20:33:37.5766366Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:33:38.6838973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:38.7059617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:38.7060510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:38.7143405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:38.7149295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:38.8071387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:39.3692494Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:39.5749645Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:41.3880851Z ok (4.423s) 2022-03-17T20:33:41.3881409Z 2022-03-17T20:33:41.3882904Z ---------------------------------------------------------------------- 2022-03-17T20:33:41.3883936Z Ran 1 test in 4.424s 2022-03-17T20:33:41.3884387Z 2022-03-17T20:33:41.3884619Z OK 2022-03-17T20:33:41.3884976Z 2022-03-17T20:33:41.3885294Z Generating XML reports... 2022-03-17T20:33:41.3964763Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203336.xml 2022-03-17T20:33:44.0673463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:44.0684458Z 2022-03-17T20:33:44.0685131Z Running tests... 2022-03-17T20:33:44.0686307Z ---------------------------------------------------------------------- 2022-03-17T20:33:44.6773046Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110538 2022-03-17T20:33:44.6816579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110539 2022-03-17T20:33:44.6817959Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:33:45.8310618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:45.8349976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:45.8350866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:45.8411900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:45.8416338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:45.9362521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:46.2292463Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:46.3297998Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:33:55.8403435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:55.8404657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:55.8425237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:55.8426454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:58.6118891Z ok (14.543s) 2022-03-17T20:33:58.6119413Z 2022-03-17T20:33:58.6120669Z ---------------------------------------------------------------------- 2022-03-17T20:33:58.6121450Z Ran 1 test in 14.543s 2022-03-17T20:33:58.6121811Z 2022-03-17T20:33:58.6122309Z OK 2022-03-17T20:33:58.6122616Z 2022-03-17T20:33:58.6122881Z Generating XML reports... 2022-03-17T20:33:58.6201296Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203344.xml 2022-03-17T20:34:01.2067729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:01.2080256Z 2022-03-17T20:34:01.2080558Z Running tests... 2022-03-17T20:34:01.2081201Z ---------------------------------------------------------------------- 2022-03-17T20:34:01.2096273Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:01.2096625Z 2022-03-17T20:34:01.2097069Z ---------------------------------------------------------------------- 2022-03-17T20:34:01.2097873Z Ran 1 test in 0.002s 2022-03-17T20:34:01.2098044Z 2022-03-17T20:34:01.2098144Z OK (skipped=1) 2022-03-17T20:34:01.2098296Z 2022-03-17T20:34:01.2098421Z Generating XML reports... 2022-03-17T20:34:01.2137903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203401.xml 2022-03-17T20:34:03.9800224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:03.9810060Z 2022-03-17T20:34:03.9810347Z Running tests... 2022-03-17T20:34:03.9811751Z ---------------------------------------------------------------------- 2022-03-17T20:34:03.9827952Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:03.9828774Z 2022-03-17T20:34:03.9829727Z ---------------------------------------------------------------------- 2022-03-17T20:34:03.9830672Z Ran 1 test in 0.002s 2022-03-17T20:34:03.9831128Z 2022-03-17T20:34:03.9831362Z OK (skipped=1) 2022-03-17T20:34:03.9831706Z 2022-03-17T20:34:03.9831972Z Generating XML reports... 2022-03-17T20:34:03.9870502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203403.xml 2022-03-17T20:34:06.7123468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:06.7135138Z 2022-03-17T20:34:06.7135739Z Running tests... 2022-03-17T20:34:06.7137177Z ---------------------------------------------------------------------- 2022-03-17T20:34:06.7154960Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:06.7155792Z 2022-03-17T20:34:06.7156755Z ---------------------------------------------------------------------- 2022-03-17T20:34:06.7157718Z Ran 1 test in 0.002s 2022-03-17T20:34:06.7158070Z 2022-03-17T20:34:06.7158301Z OK (skipped=1) 2022-03-17T20:34:06.7158650Z 2022-03-17T20:34:06.7158920Z Generating XML reports... 2022-03-17T20:34:06.7193161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203406.xml 2022-03-17T20:34:09.3936271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:09.3947808Z 2022-03-17T20:34:09.3948262Z Running tests... 2022-03-17T20:34:09.3949663Z ---------------------------------------------------------------------- 2022-03-17T20:34:09.3966922Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:09.3967727Z 2022-03-17T20:34:09.3968780Z ---------------------------------------------------------------------- 2022-03-17T20:34:09.3969623Z Ran 1 test in 0.002s 2022-03-17T20:34:09.3969992Z 2022-03-17T20:34:09.3970219Z OK (skipped=1) 2022-03-17T20:34:09.3970564Z 2022-03-17T20:34:09.3970835Z Generating XML reports... 2022-03-17T20:34:09.4006898Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203409.xml 2022-03-17T20:34:12.4457879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:12.4470105Z 2022-03-17T20:34:12.4470513Z Running tests... 2022-03-17T20:34:12.4489785Z ---------------------------------------------------------------------- 2022-03-17T20:34:12.4490380Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:12.4490757Z 2022-03-17T20:34:12.4491091Z ---------------------------------------------------------------------- 2022-03-17T20:34:12.4491436Z Ran 1 test in 0.002s 2022-03-17T20:34:12.4491605Z 2022-03-17T20:34:12.4491697Z OK (skipped=1) 2022-03-17T20:34:12.4491855Z 2022-03-17T20:34:12.4491980Z Generating XML reports... 2022-03-17T20:34:12.4529932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203412.xml 2022-03-17T20:34:15.1501361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:15.1512521Z 2022-03-17T20:34:15.1513165Z Running tests... 2022-03-17T20:34:15.1514324Z ---------------------------------------------------------------------- 2022-03-17T20:34:15.1531252Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:15.1532023Z 2022-03-17T20:34:15.1533132Z ---------------------------------------------------------------------- 2022-03-17T20:34:15.1533967Z Ran 1 test in 0.002s 2022-03-17T20:34:15.1534347Z 2022-03-17T20:34:15.1534589Z OK (skipped=1) 2022-03-17T20:34:15.1534940Z 2022-03-17T20:34:15.1535212Z Generating XML reports... 2022-03-17T20:34:15.1569972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203415.xml 2022-03-17T20:34:17.8637684Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:17.8649245Z 2022-03-17T20:34:17.8649563Z Running tests... 2022-03-17T20:34:17.8650100Z ---------------------------------------------------------------------- 2022-03-17T20:34:17.8669511Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:17.8670362Z 2022-03-17T20:34:17.8670857Z ---------------------------------------------------------------------- 2022-03-17T20:34:17.8671331Z Ran 1 test in 0.002s 2022-03-17T20:34:17.8671541Z 2022-03-17T20:34:17.8671680Z OK (skipped=1) 2022-03-17T20:34:17.8671848Z 2022-03-17T20:34:17.8671980Z Generating XML reports... 2022-03-17T20:34:17.8709818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203417.xml 2022-03-17T20:34:20.5870732Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:20.5883334Z 2022-03-17T20:34:20.5883575Z Running tests... 2022-03-17T20:34:20.5884175Z ---------------------------------------------------------------------- 2022-03-17T20:34:20.5902706Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:20.5903060Z 2022-03-17T20:34:20.5903521Z ---------------------------------------------------------------------- 2022-03-17T20:34:20.5903977Z Ran 1 test in 0.002s 2022-03-17T20:34:20.5904162Z 2022-03-17T20:34:20.5904257Z OK (skipped=1) 2022-03-17T20:34:20.5904421Z 2022-03-17T20:34:20.5904545Z Generating XML reports... 2022-03-17T20:34:20.5941253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203420.xml 2022-03-17T20:34:23.2914743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:23.2929606Z 2022-03-17T20:34:23.2929970Z Running tests... 2022-03-17T20:34:23.2930498Z ---------------------------------------------------------------------- 2022-03-17T20:34:23.2943851Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:23.2944129Z 2022-03-17T20:34:23.2944592Z ---------------------------------------------------------------------- 2022-03-17T20:34:23.2944955Z Ran 1 test in 0.002s 2022-03-17T20:34:23.2945119Z 2022-03-17T20:34:23.2945233Z OK (skipped=1) 2022-03-17T20:34:23.2945380Z 2022-03-17T20:34:23.2945506Z Generating XML reports... 2022-03-17T20:34:23.2983206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203423.xml 2022-03-17T20:34:26.5520390Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:26.5532279Z 2022-03-17T20:34:26.5532946Z Running tests... 2022-03-17T20:34:26.5534094Z ---------------------------------------------------------------------- 2022-03-17T20:34:26.5547758Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:26.5548500Z 2022-03-17T20:34:26.5549457Z ---------------------------------------------------------------------- 2022-03-17T20:34:26.5550381Z Ran 1 test in 0.002s 2022-03-17T20:34:26.5551161Z 2022-03-17T20:34:26.5551390Z OK (skipped=1) 2022-03-17T20:34:26.5551739Z 2022-03-17T20:34:26.5551994Z Generating XML reports... 2022-03-17T20:34:26.5588010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203426.xml 2022-03-17T20:34:29.3377375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:29.3388594Z 2022-03-17T20:34:29.3388780Z Running tests... 2022-03-17T20:34:29.9475260Z ---------------------------------------------------------------------- 2022-03-17T20:34:29.9476595Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111427 2022-03-17T20:34:29.9563752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111428 2022-03-17T20:34:29.9565814Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:34:31.0178447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:31.0597176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:31.0598113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:31.0685536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:31.0687160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:31.1610697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:31.5435387Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:34:31.5978838Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:34:33.4667366Z ok (4.127s) 2022-03-17T20:34:33.4669857Z 2022-03-17T20:34:33.4670794Z ---------------------------------------------------------------------- 2022-03-17T20:34:33.4671964Z Ran 1 test in 4.128s 2022-03-17T20:34:33.4672495Z 2022-03-17T20:34:33.4672683Z OK 2022-03-17T20:34:33.4673070Z 2022-03-17T20:34:33.4673473Z Generating XML reports... 2022-03-17T20:34:33.4746532Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203429.xml 2022-03-17T20:34:36.5308710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:36.5322333Z 2022-03-17T20:34:36.5323399Z Running tests... 2022-03-17T20:34:36.5324574Z ---------------------------------------------------------------------- 2022-03-17T20:34:36.5342556Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:36.5343344Z 2022-03-17T20:34:36.5344312Z ---------------------------------------------------------------------- 2022-03-17T20:34:36.5345320Z Ran 1 test in 0.002s 2022-03-17T20:34:36.5345709Z 2022-03-17T20:34:36.5345945Z OK (skipped=1) 2022-03-17T20:34:36.5346295Z 2022-03-17T20:34:36.5346565Z Generating XML reports... 2022-03-17T20:34:36.5379796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203436.xml 2022-03-17T20:34:39.2822012Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:39.2833889Z 2022-03-17T20:34:39.2834350Z Running tests... 2022-03-17T20:34:39.2835822Z ---------------------------------------------------------------------- 2022-03-17T20:34:39.2854609Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:39.2855341Z 2022-03-17T20:34:39.2856306Z ---------------------------------------------------------------------- 2022-03-17T20:34:39.2857241Z Ran 1 test in 0.002s 2022-03-17T20:34:39.2857608Z 2022-03-17T20:34:39.2858223Z OK (skipped=1) 2022-03-17T20:34:39.2858592Z 2022-03-17T20:34:39.2858865Z Generating XML reports... 2022-03-17T20:34:39.2892109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203439.xml 2022-03-17T20:34:41.9872831Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:41.9884806Z 2022-03-17T20:34:41.9885393Z Running tests... 2022-03-17T20:34:41.9886615Z ---------------------------------------------------------------------- 2022-03-17T20:34:42.6250095Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111782 2022-03-17T20:34:42.6306051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111783 2022-03-17T20:34:42.6307368Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:34:43.7003843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:43.7218085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:43.7219057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:43.7304197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:43.7306799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:43.8225697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:44.1986764Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:34:44.2494207Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:34:46.0412625Z ok (4.052s) 2022-03-17T20:34:46.0413177Z 2022-03-17T20:34:46.0414306Z ---------------------------------------------------------------------- 2022-03-17T20:34:46.0415255Z Ran 1 test in 4.053s 2022-03-17T20:34:46.0415690Z 2022-03-17T20:34:46.0415856Z OK 2022-03-17T20:34:46.0416147Z 2022-03-17T20:34:46.0416419Z Generating XML reports... 2022-03-17T20:34:46.0488454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203441.xml 2022-03-17T20:34:48.8483080Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:48.8493321Z 2022-03-17T20:34:48.8493720Z Running tests... 2022-03-17T20:34:48.8494824Z ---------------------------------------------------------------------- 2022-03-17T20:34:49.4799578Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112003 2022-03-17T20:34:49.4848191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112004 2022-03-17T20:34:49.4849592Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:34:50.5342951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:50.5481230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:50.5482086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:50.5547431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:50.5555576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:50.6489278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:50.9402852Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:34:51.0254474Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:34:53.2956875Z ok (4.446s) 2022-03-17T20:34:53.2957405Z 2022-03-17T20:34:53.2958588Z ---------------------------------------------------------------------- 2022-03-17T20:34:53.2959542Z Ran 1 test in 4.446s 2022-03-17T20:34:53.2959984Z 2022-03-17T20:34:53.2960181Z OK 2022-03-17T20:34:53.2960487Z 2022-03-17T20:34:53.2960792Z Generating XML reports... 2022-03-17T20:34:53.3042886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203448.xml 2022-03-17T20:34:56.0310891Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:56.0324834Z 2022-03-17T20:34:56.0325177Z Running tests... 2022-03-17T20:34:56.0325715Z ---------------------------------------------------------------------- 2022-03-17T20:34:56.0341083Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:56.0341491Z 2022-03-17T20:34:56.0341980Z ---------------------------------------------------------------------- 2022-03-17T20:34:56.0342339Z Ran 1 test in 0.002s 2022-03-17T20:34:56.0342504Z 2022-03-17T20:34:56.0342606Z OK (skipped=1) 2022-03-17T20:34:56.0342765Z 2022-03-17T20:34:56.0342890Z Generating XML reports... 2022-03-17T20:34:56.0379728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203456.xml 2022-03-17T20:34:58.7181492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:58.7191207Z 2022-03-17T20:34:58.7191602Z Running tests... 2022-03-17T20:34:58.7193648Z ---------------------------------------------------------------------- 2022-03-17T20:34:58.7206242Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:34:58.7206966Z 2022-03-17T20:34:58.7207935Z ---------------------------------------------------------------------- 2022-03-17T20:34:58.7208858Z Ran 1 test in 0.002s 2022-03-17T20:34:58.7209307Z 2022-03-17T20:34:58.7209542Z OK (skipped=1) 2022-03-17T20:34:58.7209913Z 2022-03-17T20:34:58.7210173Z Generating XML reports... 2022-03-17T20:34:58.7247594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203458.xml 2022-03-17T20:35:01.4165359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:01.4175218Z 2022-03-17T20:35:01.4175781Z Running tests... 2022-03-17T20:35:01.4177187Z ---------------------------------------------------------------------- 2022-03-17T20:35:01.4199830Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:01.4200769Z 2022-03-17T20:35:01.4201583Z ---------------------------------------------------------------------- 2022-03-17T20:35:01.4203253Z Ran 1 test in 0.002s 2022-03-17T20:35:01.4203728Z 2022-03-17T20:35:01.4204068Z OK (skipped=1) 2022-03-17T20:35:01.4204540Z 2022-03-17T20:35:01.4206762Z Generating XML reports... 2022-03-17T20:35:01.4240794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203501.xml 2022-03-17T20:35:04.1447066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:04.1455773Z 2022-03-17T20:35:04.1456156Z Running tests... 2022-03-17T20:35:04.1457460Z ---------------------------------------------------------------------- 2022-03-17T20:35:04.1470338Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:35:04.1471116Z 2022-03-17T20:35:04.1472089Z ---------------------------------------------------------------------- 2022-03-17T20:35:04.1473032Z Ran 1 test in 0.002s 2022-03-17T20:35:04.1473385Z 2022-03-17T20:35:04.1473620Z OK (skipped=1) 2022-03-17T20:35:04.1473975Z 2022-03-17T20:35:04.1474635Z Generating XML reports... 2022-03-17T20:35:04.1511049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203504.xml 2022-03-17T20:35:06.9280829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:06.9290889Z 2022-03-17T20:35:06.9291327Z Running tests... 2022-03-17T20:35:06.9292752Z ---------------------------------------------------------------------- 2022-03-17T20:35:07.5331955Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112492 2022-03-17T20:35:07.5379439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112493 2022-03-17T20:35:07.5380999Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:35:08.6726465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:08.6834084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:08.6836540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:08.6930133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:08.6931519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:08.7844336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:09.1677093Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:09.2177067Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:11.0479960Z ok (4.118s) 2022-03-17T20:35:11.0480731Z 2022-03-17T20:35:11.0481757Z ---------------------------------------------------------------------- 2022-03-17T20:35:11.0482709Z Ran 1 test in 4.119s 2022-03-17T20:35:11.0483051Z 2022-03-17T20:35:11.0483242Z OK 2022-03-17T20:35:11.0483527Z 2022-03-17T20:35:11.0483828Z Generating XML reports... 2022-03-17T20:35:11.0551393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203506.xml 2022-03-17T20:35:13.7106069Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:13.7115374Z 2022-03-17T20:35:13.7115787Z Running tests... 2022-03-17T20:35:13.7116926Z ---------------------------------------------------------------------- 2022-03-17T20:35:14.3171479Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112713 2022-03-17T20:35:14.3208512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112714 2022-03-17T20:35:14.3209836Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:35:15.4483281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:15.4585820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:15.4587810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:15.4685419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:15.4688718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:15.5602224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:15.8476111Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:15.9409184Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:17.8339837Z ok (4.122s) 2022-03-17T20:35:17.8340265Z 2022-03-17T20:35:17.8342159Z ---------------------------------------------------------------------- 2022-03-17T20:35:17.8343031Z Ran 1 test in 4.122s 2022-03-17T20:35:17.8343439Z 2022-03-17T20:35:17.8343638Z OK 2022-03-17T20:35:17.8343955Z 2022-03-17T20:35:17.8344246Z Generating XML reports... 2022-03-17T20:35:17.8419075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203513.xml 2022-03-17T20:35:20.5685588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:20.5695888Z 2022-03-17T20:35:20.5696338Z Running tests... 2022-03-17T20:35:20.5697507Z ---------------------------------------------------------------------- 2022-03-17T20:35:20.5712755Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:35:20.5713475Z 2022-03-17T20:35:20.5714284Z ---------------------------------------------------------------------- 2022-03-17T20:35:20.5715074Z Ran 1 test in 0.002s 2022-03-17T20:35:20.5715440Z 2022-03-17T20:35:20.5715659Z OK (skipped=1) 2022-03-17T20:35:20.5716014Z 2022-03-17T20:35:20.5716287Z Generating XML reports... 2022-03-17T20:35:20.5751960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203520.xml 2022-03-17T20:35:23.8021764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:23.8032707Z 2022-03-17T20:35:23.8033352Z Running tests... 2022-03-17T20:35:23.8034019Z ---------------------------------------------------------------------- 2022-03-17T20:35:23.8046394Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:23.8046683Z 2022-03-17T20:35:23.8048717Z ---------------------------------------------------------------------- 2022-03-17T20:35:23.8049527Z Ran 1 test in 0.002s 2022-03-17T20:35:23.8049914Z 2022-03-17T20:35:23.8050142Z OK (skipped=1) 2022-03-17T20:35:23.8050483Z 2022-03-17T20:35:23.8050753Z Generating XML reports... 2022-03-17T20:35:23.8088698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203523.xml 2022-03-17T20:35:26.8736610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:26.8749036Z 2022-03-17T20:35:26.8749393Z Running tests... 2022-03-17T20:35:26.8750565Z ---------------------------------------------------------------------- 2022-03-17T20:35:26.8771199Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:26.8771858Z 2022-03-17T20:35:26.8772644Z ---------------------------------------------------------------------- 2022-03-17T20:35:26.8773410Z Ran 1 test in 0.002s 2022-03-17T20:35:26.8773757Z 2022-03-17T20:35:26.8774004Z OK (skipped=1) 2022-03-17T20:35:26.8774357Z 2022-03-17T20:35:26.8774640Z Generating XML reports... 2022-03-17T20:35:26.8811903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203526.xml 2022-03-17T20:35:29.7879081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:29.7893626Z 2022-03-17T20:35:29.7893986Z Running tests... 2022-03-17T20:35:29.7894685Z ---------------------------------------------------------------------- 2022-03-17T20:35:29.7905644Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:35:29.7905918Z 2022-03-17T20:35:29.7906543Z ---------------------------------------------------------------------- 2022-03-17T20:35:29.7907027Z Ran 1 test in 0.002s 2022-03-17T20:35:29.7907252Z 2022-03-17T20:35:29.7907390Z OK (skipped=1) 2022-03-17T20:35:29.7907601Z 2022-03-17T20:35:29.7907723Z Generating XML reports... 2022-03-17T20:35:29.7947624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203529.xml 2022-03-17T20:35:32.5202652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:32.5214836Z 2022-03-17T20:35:32.5215402Z Running tests... 2022-03-17T20:35:32.5216590Z ---------------------------------------------------------------------- 2022-03-17T20:35:32.5232044Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:32.5232717Z 2022-03-17T20:35:32.5233523Z ---------------------------------------------------------------------- 2022-03-17T20:35:32.5234296Z Ran 1 test in 0.002s 2022-03-17T20:35:32.5234683Z 2022-03-17T20:35:32.5234914Z OK (skipped=1) 2022-03-17T20:35:32.5235273Z 2022-03-17T20:35:32.5235532Z Generating XML reports... 2022-03-17T20:35:32.5273245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203532.xml 2022-03-17T20:35:35.2446319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:35.2457759Z 2022-03-17T20:35:35.2457973Z Running tests... 2022-03-17T20:35:35.2458515Z ---------------------------------------------------------------------- 2022-03-17T20:35:35.2473660Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:35:35.2473987Z 2022-03-17T20:35:35.2474448Z ---------------------------------------------------------------------- 2022-03-17T20:35:35.2474935Z Ran 1 test in 0.002s 2022-03-17T20:35:35.2475095Z 2022-03-17T20:35:35.2475201Z OK (skipped=1) 2022-03-17T20:35:35.2475365Z 2022-03-17T20:35:35.2475497Z Generating XML reports... 2022-03-17T20:35:35.2511567Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203535.xml 2022-03-17T20:35:37.9553486Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:37.9565075Z 2022-03-17T20:35:37.9565522Z Running tests... 2022-03-17T20:35:37.9566811Z ---------------------------------------------------------------------- 2022-03-17T20:35:37.9582125Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:37.9583038Z 2022-03-17T20:35:37.9584068Z ---------------------------------------------------------------------- 2022-03-17T20:35:37.9584879Z Ran 1 test in 0.002s 2022-03-17T20:35:37.9585758Z 2022-03-17T20:35:37.9586019Z OK (skipped=1) 2022-03-17T20:35:37.9586384Z 2022-03-17T20:35:37.9586661Z Generating XML reports... 2022-03-17T20:35:37.9620515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203537.xml 2022-03-17T20:35:40.6547177Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:40.6557980Z 2022-03-17T20:35:40.6558319Z Running tests... 2022-03-17T20:35:40.6559696Z ---------------------------------------------------------------------- 2022-03-17T20:35:40.6573673Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:35:40.6574499Z 2022-03-17T20:35:40.6575382Z ---------------------------------------------------------------------- 2022-03-17T20:35:40.6576261Z Ran 1 test in 0.002s 2022-03-17T20:35:40.6576708Z 2022-03-17T20:35:40.6576975Z OK (skipped=1) 2022-03-17T20:35:40.6577382Z 2022-03-17T20:35:40.6577715Z Generating XML reports... 2022-03-17T20:35:40.6614931Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203540.xml 2022-03-17T20:35:43.6341686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:43.6353110Z 2022-03-17T20:35:43.6353437Z Running tests... 2022-03-17T20:35:43.6354737Z ---------------------------------------------------------------------- 2022-03-17T20:35:44.2422221Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113470 2022-03-17T20:35:44.2518745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113471 2022-03-17T20:35:44.2520168Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:35:45.3837949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:45.3963765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:45.3965661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:45.4040379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:45.4042691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:45.4975610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:45.8743172Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:45.9221390Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:48.4624572Z ok (4.827s) 2022-03-17T20:35:48.4625057Z 2022-03-17T20:35:48.4626201Z ---------------------------------------------------------------------- 2022-03-17T20:35:48.4627217Z Ran 1 test in 4.827s 2022-03-17T20:35:48.4627656Z 2022-03-17T20:35:48.4627896Z OK 2022-03-17T20:35:48.4628275Z 2022-03-17T20:35:48.4628623Z Generating XML reports... 2022-03-17T20:35:48.4705304Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203543.xml 2022-03-17T20:35:51.5243176Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:51.5252975Z 2022-03-17T20:35:51.5253381Z Running tests... 2022-03-17T20:35:51.5254771Z ---------------------------------------------------------------------- 2022-03-17T20:35:52.1310253Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113685 2022-03-17T20:35:52.1355084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113686 2022-03-17T20:35:52.1357107Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:35:53.2349350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:53.2580128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:53.2582227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:53.2654021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:53.2655734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:53.3596458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:53.7464411Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:53.7979070Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:35:55.6450879Z ok (4.119s) 2022-03-17T20:35:55.6451360Z 2022-03-17T20:35:55.6452486Z ---------------------------------------------------------------------- 2022-03-17T20:35:55.6453385Z Ran 1 test in 4.120s 2022-03-17T20:35:55.6453790Z 2022-03-17T20:35:55.6454004Z OK 2022-03-17T20:35:55.6454310Z 2022-03-17T20:35:55.6454628Z Generating XML reports... 2022-03-17T20:35:55.6532474Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203551.xml 2022-03-17T20:35:58.2984279Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:58.2995047Z 2022-03-17T20:35:58.2995446Z Running tests... 2022-03-17T20:35:58.2996802Z ---------------------------------------------------------------------- 2022-03-17T20:35:58.9063822Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113906 2022-03-17T20:35:58.9110702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113907 2022-03-17T20:35:58.9112240Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:35:59.9587485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:00.0520633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:00.0523488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:00.0599260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:00.0600852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:00.1534013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:00.4382633Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:00.5373299Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:02.4215354Z ok (4.121s) 2022-03-17T20:36:02.4215879Z 2022-03-17T20:36:02.4217169Z ---------------------------------------------------------------------- 2022-03-17T20:36:02.4218353Z Ran 1 test in 4.122s 2022-03-17T20:36:02.4218724Z 2022-03-17T20:36:02.4218910Z OK 2022-03-17T20:36:02.4219202Z 2022-03-17T20:36:02.4219486Z Generating XML reports... 2022-03-17T20:36:02.4298610Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203558.xml 2022-03-17T20:36:05.0251571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:05.0263922Z 2022-03-17T20:36:05.0264152Z Running tests... 2022-03-17T20:36:05.0264631Z ---------------------------------------------------------------------- 2022-03-17T20:36:05.0280080Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:05.0280409Z 2022-03-17T20:36:05.0280783Z ---------------------------------------------------------------------- 2022-03-17T20:36:05.0281127Z Ran 1 test in 0.002s 2022-03-17T20:36:05.0281299Z 2022-03-17T20:36:05.0281429Z OK (skipped=1) 2022-03-17T20:36:05.0281595Z 2022-03-17T20:36:05.0281719Z Generating XML reports... 2022-03-17T20:36:05.0318438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203605.xml 2022-03-17T20:36:08.3567984Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:08.3573947Z 2022-03-17T20:36:08.3574208Z Running tests... 2022-03-17T20:36:08.3575639Z ---------------------------------------------------------------------- 2022-03-17T20:36:08.3594360Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:08.3594999Z 2022-03-17T20:36:08.3596223Z ---------------------------------------------------------------------- 2022-03-17T20:36:08.3596979Z Ran 1 test in 0.002s 2022-03-17T20:36:08.3597349Z 2022-03-17T20:36:08.3597596Z OK (skipped=1) 2022-03-17T20:36:08.3597954Z 2022-03-17T20:36:08.3598229Z Generating XML reports... 2022-03-17T20:36:08.3632172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203608.xml 2022-03-17T20:36:11.0397865Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:11.0408472Z 2022-03-17T20:36:11.0409007Z Running tests... 2022-03-17T20:36:11.0410168Z ---------------------------------------------------------------------- 2022-03-17T20:36:11.0423425Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:36:11.0424042Z 2022-03-17T20:36:11.0424803Z ---------------------------------------------------------------------- 2022-03-17T20:36:11.0425571Z Ran 1 test in 0.002s 2022-03-17T20:36:11.0425943Z 2022-03-17T20:36:11.0426177Z OK (skipped=1) 2022-03-17T20:36:11.0426508Z 2022-03-17T20:36:11.0426780Z Generating XML reports... 2022-03-17T20:36:11.0463666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203611.xml 2022-03-17T20:36:13.9389491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:13.9399581Z 2022-03-17T20:36:13.9400021Z Running tests... 2022-03-17T20:36:13.9401007Z ---------------------------------------------------------------------- 2022-03-17T20:36:13.9417956Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:13.9418635Z 2022-03-17T20:36:13.9419459Z ---------------------------------------------------------------------- 2022-03-17T20:36:13.9420249Z Ran 1 test in 0.002s 2022-03-17T20:36:13.9420620Z 2022-03-17T20:36:13.9420832Z OK (skipped=1) 2022-03-17T20:36:13.9421185Z 2022-03-17T20:36:13.9421455Z Generating XML reports... 2022-03-17T20:36:13.9455672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203613.xml 2022-03-17T20:36:16.6559436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:16.6571348Z 2022-03-17T20:36:16.6571825Z Running tests... 2022-03-17T20:36:16.6573008Z ---------------------------------------------------------------------- 2022-03-17T20:36:16.6587210Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:16.6587865Z 2022-03-17T20:36:16.6588788Z ---------------------------------------------------------------------- 2022-03-17T20:36:16.6589571Z Ran 1 test in 0.002s 2022-03-17T20:36:16.6590383Z 2022-03-17T20:36:16.6590607Z OK (skipped=1) 2022-03-17T20:36:16.6590958Z 2022-03-17T20:36:16.6591229Z Generating XML reports... 2022-03-17T20:36:16.6626107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203616.xml 2022-03-17T20:36:19.4205854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:19.4216928Z 2022-03-17T20:36:19.4217437Z Running tests... 2022-03-17T20:36:19.4218699Z ---------------------------------------------------------------------- 2022-03-17T20:36:19.4232649Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:36:19.4233570Z 2022-03-17T20:36:19.4234593Z ---------------------------------------------------------------------- 2022-03-17T20:36:19.4235440Z Ran 1 test in 0.002s 2022-03-17T20:36:19.4235812Z 2022-03-17T20:36:19.4236048Z OK (skipped=1) 2022-03-17T20:36:19.4236417Z 2022-03-17T20:36:19.4236699Z Generating XML reports... 2022-03-17T20:36:19.4271477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203619.xml 2022-03-17T20:36:22.0804047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:22.0814238Z 2022-03-17T20:36:22.0814779Z Running tests... 2022-03-17T20:36:22.0815958Z ---------------------------------------------------------------------- 2022-03-17T20:36:22.6887888Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114529 2022-03-17T20:36:22.6989888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114530 2022-03-17T20:36:22.6991200Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:36:23.8424261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:23.8479992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:23.8482478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:23.8524877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:23.8528191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:23.9497595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:24.2292804Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:24.3295610Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:36.1316606Z ok (14.050s) 2022-03-17T20:36:36.1317094Z 2022-03-17T20:36:36.1318371Z ---------------------------------------------------------------------- 2022-03-17T20:36:36.1319203Z Ran 1 test in 14.050s 2022-03-17T20:36:36.1319568Z 2022-03-17T20:36:36.1319737Z OK 2022-03-17T20:36:36.1320028Z 2022-03-17T20:36:36.1320315Z Generating XML reports... 2022-03-17T20:36:36.1392364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203622.xml 2022-03-17T20:36:39.1510509Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:39.1522214Z 2022-03-17T20:36:39.1522566Z Running tests... 2022-03-17T20:36:39.1523161Z ---------------------------------------------------------------------- 2022-03-17T20:36:39.1542292Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:39.1542734Z 2022-03-17T20:36:39.1543202Z ---------------------------------------------------------------------- 2022-03-17T20:36:39.1543637Z Ran 1 test in 0.002s 2022-03-17T20:36:39.1544142Z 2022-03-17T20:36:39.1544240Z OK (skipped=1) 2022-03-17T20:36:39.1544405Z 2022-03-17T20:36:39.1544533Z Generating XML reports... 2022-03-17T20:36:39.1579672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203639.xml 2022-03-17T20:36:41.8759156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:41.8770138Z 2022-03-17T20:36:41.8770971Z Running tests... 2022-03-17T20:36:41.8773509Z ---------------------------------------------------------------------- 2022-03-17T20:36:42.4822889Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114815 2022-03-17T20:36:42.4873027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114816 2022-03-17T20:36:42.4874403Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:36:43.5823307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:43.6399247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:43.6401284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:43.6429434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:43.6431970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:43.7411833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:44.1237130Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:44.1738338Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:46.1986679Z ok (4.321s) 2022-03-17T20:36:46.1987065Z 2022-03-17T20:36:46.1988722Z ---------------------------------------------------------------------- 2022-03-17T20:36:46.1989496Z Ran 1 test in 4.322s 2022-03-17T20:36:46.1989886Z 2022-03-17T20:36:46.1990079Z OK 2022-03-17T20:36:46.1990375Z 2022-03-17T20:36:46.1990654Z Generating XML reports... 2022-03-17T20:36:46.2061436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203641.xml 2022-03-17T20:36:49.0648653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:49.0660506Z 2022-03-17T20:36:49.0660728Z Running tests... 2022-03-17T20:36:49.0661217Z ---------------------------------------------------------------------- 2022-03-17T20:36:49.6726379Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115030 2022-03-17T20:36:49.6825338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115031 2022-03-17T20:36:49.6826739Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:36:50.7797074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:50.8019383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:50.8020660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:50.8100872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:50.8111266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:50.9028862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:51.1965378Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:36:51.2801566Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:37:00.3538986Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:37:00.3607459Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:37:03.0102421Z ok (13.943s) 2022-03-17T20:37:03.0103084Z 2022-03-17T20:37:03.0104363Z ---------------------------------------------------------------------- 2022-03-17T20:37:03.0105920Z Ran 1 test in 13.944s 2022-03-17T20:37:03.0106311Z 2022-03-17T20:37:03.0106511Z OK 2022-03-17T20:37:03.0106797Z 2022-03-17T20:37:03.0108000Z Generating XML reports... 2022-03-17T20:37:03.0182223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203649.xml 2022-03-17T20:37:06.0187032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:06.0198002Z 2022-03-17T20:37:06.0198574Z Running tests... 2022-03-17T20:37:06.0199808Z ---------------------------------------------------------------------- 2022-03-17T20:37:06.6272121Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115249 2022-03-17T20:37:06.6318913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115250 2022-03-17T20:37:06.6320300Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:37:07.7444892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:07.7748773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:07.7750866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:07.7850574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:07.7858538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:07.8758500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:09.6416999Z skip (3.621s) 2022-03-17T20:37:09.6417405Z 2022-03-17T20:37:09.6418483Z ---------------------------------------------------------------------- 2022-03-17T20:37:09.6419310Z Ran 1 test in 3.622s 2022-03-17T20:37:09.6419679Z 2022-03-17T20:37:09.6419932Z OK (skipped=1) 2022-03-17T20:37:09.6420260Z 2022-03-17T20:37:09.6420530Z Generating XML reports... 2022-03-17T20:37:09.6501080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203706.xml 2022-03-17T20:37:12.2959504Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:12.2967844Z 2022-03-17T20:37:12.2968240Z Running tests... 2022-03-17T20:37:12.2969277Z ---------------------------------------------------------------------- 2022-03-17T20:37:12.9042306Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115457 2022-03-17T20:37:12.9087234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115458 2022-03-17T20:37:12.9088583Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:37:13.9867264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:14.0286113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:14.0286922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:14.0373801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:14.0377225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:14.1298696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:15.8183105Z skip (3.521s) 2022-03-17T20:37:15.8183511Z 2022-03-17T20:37:15.8185176Z ---------------------------------------------------------------------- 2022-03-17T20:37:15.8185965Z Ran 1 test in 3.521s 2022-03-17T20:37:15.8186348Z 2022-03-17T20:37:15.8186581Z OK (skipped=1) 2022-03-17T20:37:15.8186940Z 2022-03-17T20:37:15.8187191Z Generating XML reports... 2022-03-17T20:37:15.8266493Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203712.xml 2022-03-17T20:37:18.8049524Z Running distributed tests for the nccl backend with file init_method 2022-03-17T20:37:18.8051990Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:37:18.804513] 2022-03-17T20:37:19.9175243Z 2022-03-17T20:37:19.9266744Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T20:37:19.9314282Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9314752Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9315151Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9315535Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9315948Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9316384Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9316832Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9317297Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9317771Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9318282Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9318809Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9319293Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9319860Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9320331Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9320779Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9321246Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9321667Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9322119Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9322510Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9322928Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9323380Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9323820Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9324206Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9324566Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9324971Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9325372Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9325752Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9326156Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9326530Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9326895Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9327260Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9327629Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9327999Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9328353Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9328737Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9329124Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9329575Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9329978Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9330379Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9330805Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9331224Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9331619Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9332020Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9332434Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9333001Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9333494Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9333934Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9334354Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9334736Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9335127Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9335528Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9335921Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9336383Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9336756Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9337145Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9337521Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9337878Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9338259Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9338622Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9338982Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9339333Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9339686Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9340069Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9340454Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9340820Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9341163Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9341521Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9341891Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9342255Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9342629Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9343005Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9343363Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9343716Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9344058Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9344434Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9344791Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9345163Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9345537Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9345892Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9346314Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9346713Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9347126Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9347546Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9347958Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9348387Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9348800Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9349216Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9349629Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9350029Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9350448Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9350863Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9351295Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9351725Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9352137Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9352611Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9353005Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9353372Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9353726Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9354061Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9354403Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9354755Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9355123Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9355477Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9355837Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9356221Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9356608Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9356994Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9357362Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9357731Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9358143Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9358527Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9358910Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9359305Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9359681Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9360071Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9360444Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9360815Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9361165Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9361520Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9361944Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9362299Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9362725Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9363154Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9363620Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9364054Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9364427Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9364822Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9365228Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9365634Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9366075Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9366484Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9366878Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9367293Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9367660Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9368009Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9368364Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9368798Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9369165Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9369555Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9369965Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9370367Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9370761Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9371182Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9371657Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9372193Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9372765Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9373339Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9373913Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9374480Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9375061Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9375648Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9376216Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9376729Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9377194Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9377603Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9378011Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9378377Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9378759Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9379128Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9379512Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9379918Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9380341Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9380781Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9381173Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9381526Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9381916Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9382313Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9382717Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9383094Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9383486Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9383884Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9384325Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9384721Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9385101Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9385489Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9385884Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9386251Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9386648Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9387067Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9387475Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9387831Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9388198Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9388593Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9388972Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9389332Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9389672Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9390011Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9390361Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9390702Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9391051Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9391415Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9391770Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9392110Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9392432Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9392790Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9393148Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9393518Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9393865Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9394238Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9394595Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9394963Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9395354Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9395785Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9396190Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9396584Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9396968Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9397378Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9397779Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9398170Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9398563Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9398940Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9399328Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9399711Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9400068Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9400492Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9400921Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9401382Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9401943Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9402360Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9402788Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9403197Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9403603Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9403996Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9404381Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9404777Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9405136Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9405508Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9405886Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9406235Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9406593Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9406952Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9407312Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9407649Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9407990Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9408335Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9408679Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9409028Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9409368Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9409733Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9410093Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9410420Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9410838Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9411186Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9411537Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9411899Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9412252Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9412611Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9412961Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9413309Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9413647Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9414040Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9414462Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9414857Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9415228Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9415594Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9415991Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9416379Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9416734Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9417178Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9417577Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9417967Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9418344Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9418712Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9419090Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9419433Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9419827Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9420248Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:19.9420649Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:37:20.9636249Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:20.9648355Z 2022-03-17T20:37:20.9648894Z Running tests... 2022-03-17T20:37:20.9649428Z ---------------------------------------------------------------------- 2022-03-17T20:37:21.5673942Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115732 2022-03-17T20:37:21.5712479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115733 2022-03-17T20:37:21.5716245Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:37:22.6390089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:22.6748160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:22.6750202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:22.6795934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:22.6799905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:22.7760941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:23.0769919Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.0772390Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.1732467Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.1734935Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.3092303Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:37:23.5403497Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:37:23.6326712Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.6329507Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.6331396Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.6333858Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.6425682Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.6426905Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.6427563Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.6428361Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.6522350Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.6524743Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:23.6526503Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:37:23.6528617Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:37:25.3817306Z ok (4.416s) 2022-03-17T20:37:25.3817700Z 2022-03-17T20:37:25.3818878Z ---------------------------------------------------------------------- 2022-03-17T20:37:25.3819783Z Ran 1 test in 4.417s 2022-03-17T20:37:25.3820196Z 2022-03-17T20:37:25.3820390Z OK 2022-03-17T20:37:25.3821799Z 2022-03-17T20:37:25.3822302Z Generating XML reports... 2022-03-17T20:37:25.3852967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203720.xml 2022-03-17T20:37:27.9180187Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:27.9192749Z 2022-03-17T20:37:27.9193032Z Running tests... 2022-03-17T20:37:27.9193572Z ---------------------------------------------------------------------- 2022-03-17T20:37:27.9228460Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T20:37:27.9228797Z 2022-03-17T20:37:27.9229168Z ---------------------------------------------------------------------- 2022-03-17T20:37:27.9229720Z Ran 1 test in 0.004s 2022-03-17T20:37:27.9229884Z 2022-03-17T20:37:27.9229994Z OK (skipped=1) 2022-03-17T20:37:27.9230154Z 2022-03-17T20:37:27.9230293Z Generating XML reports... 2022-03-17T20:37:27.9266331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203727.xml 2022-03-17T20:37:30.5945012Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:30.5958024Z 2022-03-17T20:37:30.5958343Z Running tests... 2022-03-17T20:37:30.5958891Z ---------------------------------------------------------------------- 2022-03-17T20:37:31.2035604Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116014 2022-03-17T20:37:31.2075142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116015 2022-03-17T20:37:31.2076499Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:37:32.3032324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:32.3335774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:32.3337692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:32.3435978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:32.3438355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:32.4350312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:34.2148752Z ok (3.618s) 2022-03-17T20:37:34.2149167Z 2022-03-17T20:37:34.2150347Z ---------------------------------------------------------------------- 2022-03-17T20:37:34.2151106Z Ran 1 test in 3.619s 2022-03-17T20:37:34.2151479Z 2022-03-17T20:37:34.2151668Z OK 2022-03-17T20:37:34.2151963Z 2022-03-17T20:37:34.2152271Z Generating XML reports... 2022-03-17T20:37:34.2224863Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203730.xml 2022-03-17T20:37:36.8581116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:36.8592538Z 2022-03-17T20:37:36.8592815Z Running tests... 2022-03-17T20:37:36.8593340Z ---------------------------------------------------------------------- 2022-03-17T20:37:37.4591122Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116222 2022-03-17T20:37:37.4624876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116223 2022-03-17T20:37:37.4626238Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:37:38.5833320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:38.5953678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:38.5955910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:38.6037473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:38.6042447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:38.6964008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:38.9877766Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:37:39.0826954Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:37:48.4937646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.5377800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.5561653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.5564017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.5631388Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:37:48.5633437Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:37:48.5652602Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:37:48.5654643Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:37:48.6905993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.6907246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.8281329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.8282694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.8491416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.8494532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.9756705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.9757917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.9950204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:48.9951455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.1269465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.1270738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.2586947Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:37:49.2588461Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:37:49.2590531Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:37:49.2591946Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:37:49.2773167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.2774421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.4219135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.4224838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.5690897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.5718739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.6078027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.6079269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.6276926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.6278134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.7539356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.7540575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.8911983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.8913241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.9103363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.9104581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.9309534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.9310775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.9508189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:49.9509731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.0793391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.0794626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.2182132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.2183860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.3623400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.3624611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.5047699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:50.5048930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:37:53.1958284Z ok (16.336s) 2022-03-17T20:37:53.1958763Z 2022-03-17T20:37:53.1959876Z ---------------------------------------------------------------------- 2022-03-17T20:37:53.1960797Z Ran 1 test in 16.336s 2022-03-17T20:37:53.1961232Z 2022-03-17T20:37:53.1961464Z OK 2022-03-17T20:37:53.1962271Z 2022-03-17T20:37:53.1962596Z Generating XML reports... 2022-03-17T20:37:53.2045561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203736.xml 2022-03-17T20:37:56.0476792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:56.0487895Z 2022-03-17T20:37:56.0488318Z Running tests... 2022-03-17T20:37:56.0489720Z ---------------------------------------------------------------------- 2022-03-17T20:37:56.0503293Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:37:56.0504171Z 2022-03-17T20:37:56.0505141Z ---------------------------------------------------------------------- 2022-03-17T20:37:56.0506027Z Ran 1 test in 0.001s 2022-03-17T20:37:56.0506395Z 2022-03-17T20:37:56.0506624Z OK (skipped=1) 2022-03-17T20:37:56.0506976Z 2022-03-17T20:37:56.0507229Z Generating XML reports... 2022-03-17T20:37:56.0542128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203756.xml 2022-03-17T20:37:59.0219782Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:59.0229670Z 2022-03-17T20:37:59.0230103Z Running tests... 2022-03-17T20:37:59.0231497Z ---------------------------------------------------------------------- 2022-03-17T20:37:59.0243557Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:37:59.0244459Z 2022-03-17T20:37:59.0245431Z ---------------------------------------------------------------------- 2022-03-17T20:37:59.0246398Z Ran 1 test in 0.001s 2022-03-17T20:37:59.0246853Z 2022-03-17T20:37:59.0247125Z OK (skipped=1) 2022-03-17T20:37:59.0247483Z 2022-03-17T20:37:59.0247744Z Generating XML reports... 2022-03-17T20:37:59.0282755Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203759.xml 2022-03-17T20:38:01.4881681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:01.4892130Z 2022-03-17T20:38:01.4892683Z Running tests... 2022-03-17T20:38:01.4894081Z ---------------------------------------------------------------------- 2022-03-17T20:38:02.0891549Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116575 2022-03-17T20:38:02.0919185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116576 2022-03-17T20:38:02.0920772Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:03.2179949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:03.2262556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:03.2264952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:03.2279573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:03.2281187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:03.3273911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:03.6093801Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:03.7080075Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:13.2046498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.2047741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.2281396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.2294738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.2383028Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:13.2385156Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:13.2417287Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:13.2419294Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:13.3619180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.3627136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.3895990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.3908002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.5209032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.5210827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.5479113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:13.5491450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:16.4156723Z ok (14.926s) 2022-03-17T20:38:16.4157116Z 2022-03-17T20:38:16.4158126Z ---------------------------------------------------------------------- 2022-03-17T20:38:16.4158906Z Ran 1 test in 14.926s 2022-03-17T20:38:16.4159270Z 2022-03-17T20:38:16.4159456Z OK 2022-03-17T20:38:16.4159749Z 2022-03-17T20:38:16.4160012Z Generating XML reports... 2022-03-17T20:38:16.4229075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203801.xml 2022-03-17T20:38:18.9774852Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:18.9784049Z 2022-03-17T20:38:18.9784677Z Running tests... 2022-03-17T20:38:18.9785956Z ---------------------------------------------------------------------- 2022-03-17T20:38:19.5841036Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116802 2022-03-17T20:38:19.5872409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116803 2022-03-17T20:38:19.5873853Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:20.6863011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:20.7183189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:20.7185357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:20.7271895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:20.7283540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:20.8189589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:21.1112288Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:21.1972724Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:21.2858674Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:21.2870673Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:21.5468995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:21.5470424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:21.5628720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:21.5629982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:21.5697017Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:21.5699150Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:21.5716154Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:21.5718236Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:23.1959132Z ok (4.217s) 2022-03-17T20:38:23.1959545Z 2022-03-17T20:38:23.1960467Z ---------------------------------------------------------------------- 2022-03-17T20:38:23.1961273Z Ran 1 test in 4.217s 2022-03-17T20:38:23.1962039Z 2022-03-17T20:38:23.1962310Z OK 2022-03-17T20:38:23.1966132Z 2022-03-17T20:38:23.1966760Z Generating XML reports... 2022-03-17T20:38:23.2049675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203818.xml 2022-03-17T20:38:25.8389443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:25.8401098Z 2022-03-17T20:38:25.8401609Z Running tests... 2022-03-17T20:38:25.8403318Z ---------------------------------------------------------------------- 2022-03-17T20:38:26.4382177Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117021 2022-03-17T20:38:26.4418732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117022 2022-03-17T20:38:26.4420087Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:27.4957798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:27.5122187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:27.5124122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:27.5162579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:27.5163906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:27.6133191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:27.8942888Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:27.9971291Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:28.3460329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:28.3469698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:28.3629124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:28.3630449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:28.3685373Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:28.3688004Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:28.3690965Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:28.3693002Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:30.0511094Z ok (4.211s) 2022-03-17T20:38:30.0511471Z 2022-03-17T20:38:30.0512470Z ---------------------------------------------------------------------- 2022-03-17T20:38:30.0513261Z Ran 1 test in 4.211s 2022-03-17T20:38:30.0513628Z 2022-03-17T20:38:30.0513817Z OK 2022-03-17T20:38:30.0514123Z 2022-03-17T20:38:30.0514390Z Generating XML reports... 2022-03-17T20:38:30.0588512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203825.xml 2022-03-17T20:38:32.5928764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:32.5940966Z 2022-03-17T20:38:32.5941171Z Running tests... 2022-03-17T20:38:32.5942528Z ---------------------------------------------------------------------- 2022-03-17T20:38:33.1954828Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117240 2022-03-17T20:38:33.1984816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117241 2022-03-17T20:38:33.1986119Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:34.2855928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:34.3103822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:34.3106146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:34.3160645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:34.3162471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:34.4110818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:34.6925258Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:34.7895944Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:35.0102641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:35.0118616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:37.0074393Z ok (4.413s) 2022-03-17T20:38:37.0074847Z 2022-03-17T20:38:37.0075943Z ---------------------------------------------------------------------- 2022-03-17T20:38:37.0076891Z Ran 1 test in 4.413s 2022-03-17T20:38:37.0077359Z 2022-03-17T20:38:37.0077573Z OK 2022-03-17T20:38:37.0077934Z 2022-03-17T20:38:37.0078344Z Generating XML reports... 2022-03-17T20:38:37.0151217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203832.xml 2022-03-17T20:38:39.5583443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:39.5597761Z 2022-03-17T20:38:39.5598063Z Running tests... 2022-03-17T20:38:39.5598729Z ---------------------------------------------------------------------- 2022-03-17T20:38:40.1603741Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117459 2022-03-17T20:38:40.1638119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117460 2022-03-17T20:38:40.1639673Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:41.2326962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:41.2883602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:41.2884497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:41.2933957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:41.2938608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:41.3895853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:41.6726722Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:41.7691827Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:51.4062463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:51.4063789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:51.4350635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:51.4351934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:51.4437987Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:51.4440129Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:51.4492508Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:38:51.4494651Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:38:54.2980000Z ok (14.738s) 2022-03-17T20:38:54.2980409Z 2022-03-17T20:38:54.2981411Z ---------------------------------------------------------------------- 2022-03-17T20:38:54.2982185Z Ran 1 test in 14.738s 2022-03-17T20:38:54.2982568Z 2022-03-17T20:38:54.2983271Z OK 2022-03-17T20:38:54.2983655Z 2022-03-17T20:38:54.2983983Z Generating XML reports... 2022-03-17T20:38:54.3051002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203839.xml 2022-03-17T20:38:57.1891095Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:57.1900741Z 2022-03-17T20:38:57.1901332Z Running tests... 2022-03-17T20:38:57.1902676Z ---------------------------------------------------------------------- 2022-03-17T20:38:57.8175588Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117686 2022-03-17T20:38:57.8215975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117687 2022-03-17T20:38:57.8217529Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:38:58.8991226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:58.8991817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:58.8992763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:58.8993569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:58.8996137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:58.8997355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:59.2842628Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:38:59.3401975Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:08.6225579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:08.6227125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:08.6449152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:08.6457255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:08.6555327Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:39:08.6557863Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:39:08.6569590Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:39:08.6571669Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:39:11.4558031Z ok (14.265s) 2022-03-17T20:39:11.4558606Z 2022-03-17T20:39:11.4559962Z ---------------------------------------------------------------------- 2022-03-17T20:39:11.4560807Z Ran 1 test in 14.266s 2022-03-17T20:39:11.4561190Z 2022-03-17T20:39:11.4561367Z OK 2022-03-17T20:39:11.4561692Z 2022-03-17T20:39:11.4562161Z Generating XML reports... 2022-03-17T20:39:11.4632801Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203857.xml 2022-03-17T20:39:14.1711156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:14.1723087Z 2022-03-17T20:39:14.1723816Z Running tests... 2022-03-17T20:39:14.1725370Z ---------------------------------------------------------------------- 2022-03-17T20:39:14.7742109Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117905 2022-03-17T20:39:14.7769821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117906 2022-03-17T20:39:14.7771525Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:15.8957189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:15.9227465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:15.9229685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:15.9258435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:15.9261602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:16.0239912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:16.4040119Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:16.4570922Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:16.5499805Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:16.5501617Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:16.7964195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:16.8381366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:16.8544671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:16.8550467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:16.8607810Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:39:16.8610377Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:39:16.8619920Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:39:16.8621939Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:39:18.4866689Z ok (4.314s) 2022-03-17T20:39:18.4867264Z 2022-03-17T20:39:18.4868410Z ---------------------------------------------------------------------- 2022-03-17T20:39:18.4869405Z Ran 1 test in 4.314s 2022-03-17T20:39:18.4869818Z 2022-03-17T20:39:18.4870013Z OK 2022-03-17T20:39:18.4870318Z 2022-03-17T20:39:18.4870605Z Generating XML reports... 2022-03-17T20:39:18.4944078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203914.xml 2022-03-17T20:39:21.1859564Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:21.1870345Z 2022-03-17T20:39:21.1870675Z Running tests... 2022-03-17T20:39:21.1871203Z ---------------------------------------------------------------------- 2022-03-17T20:39:21.7945914Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118124 2022-03-17T20:39:21.7988744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118125 2022-03-17T20:39:21.7990072Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:22.8624514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:22.8745554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:22.8747459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:22.8827331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:22.8836374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:22.9755944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:25.1075316Z skip (3.920s) 2022-03-17T20:39:25.1075867Z 2022-03-17T20:39:25.1076933Z ---------------------------------------------------------------------- 2022-03-17T20:39:25.1077962Z Ran 1 test in 3.920s 2022-03-17T20:39:25.1078543Z 2022-03-17T20:39:25.1078780Z OK (skipped=1) 2022-03-17T20:39:25.1079145Z 2022-03-17T20:39:25.1079398Z Generating XML reports... 2022-03-17T20:39:25.1165521Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203921.xml 2022-03-17T20:39:27.7383529Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:27.7392059Z 2022-03-17T20:39:27.7392471Z Running tests... 2022-03-17T20:39:27.7393549Z ---------------------------------------------------------------------- 2022-03-17T20:39:28.3449640Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118332 2022-03-17T20:39:28.3490815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118333 2022-03-17T20:39:28.3492123Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:29.4331276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:29.4583394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:29.4585619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:29.4637992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:29.4652322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:29.5594887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:31.4585549Z ok (3.718s) 2022-03-17T20:39:31.4585925Z 2022-03-17T20:39:31.4586776Z ---------------------------------------------------------------------- 2022-03-17T20:39:31.4587568Z Ran 1 test in 3.719s 2022-03-17T20:39:31.4587932Z 2022-03-17T20:39:31.4588100Z OK 2022-03-17T20:39:31.4588410Z 2022-03-17T20:39:31.4588688Z Generating XML reports... 2022-03-17T20:39:31.4666634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203927.xml 2022-03-17T20:39:34.1602592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:34.1611824Z 2022-03-17T20:39:34.1611953Z Running tests... 2022-03-17T20:39:34.1612751Z ---------------------------------------------------------------------- 2022-03-17T20:39:34.7614604Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118540 2022-03-17T20:39:34.7657460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118541 2022-03-17T20:39:34.7658823Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:35.8364326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:35.8734770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:35.8736113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:35.8770292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:35.8771035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:35.9746793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:36.3551707Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:36.4064725Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:45.8370390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:45.8641227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:45.9284282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:45.9288219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:48.6925179Z ok (14.531s) 2022-03-17T20:39:48.6925573Z 2022-03-17T20:39:48.6926506Z ---------------------------------------------------------------------- 2022-03-17T20:39:48.6927290Z Ran 1 test in 14.531s 2022-03-17T20:39:48.6927646Z 2022-03-17T20:39:48.6927812Z OK 2022-03-17T20:39:48.6928098Z 2022-03-17T20:39:48.6928369Z Generating XML reports... 2022-03-17T20:39:48.7009922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203934.xml 2022-03-17T20:39:51.2645420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:51.2657685Z 2022-03-17T20:39:51.2658258Z Running tests... 2022-03-17T20:39:51.2659554Z ---------------------------------------------------------------------- 2022-03-17T20:39:51.8668763Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118759 2022-03-17T20:39:51.8702017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118760 2022-03-17T20:39:51.8703322Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:39:52.9361792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:53.0140153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:53.0141051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:53.0171526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:53.0172172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:53.1150583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:53.5418094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:53.5760760Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:39:55.2799928Z ok (4.014s) 2022-03-17T20:39:55.2800294Z 2022-03-17T20:39:55.2802471Z ---------------------------------------------------------------------- 2022-03-17T20:39:55.2803594Z Ran 1 test in 4.014s 2022-03-17T20:39:55.2803998Z 2022-03-17T20:39:55.2804209Z OK 2022-03-17T20:39:55.2804516Z 2022-03-17T20:39:55.2804803Z Generating XML reports... 2022-03-17T20:39:55.2885530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203951.xml 2022-03-17T20:39:57.8746506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:57.8758358Z 2022-03-17T20:39:57.8758755Z Running tests... 2022-03-17T20:39:57.8759668Z ---------------------------------------------------------------------- 2022-03-17T20:39:57.8774437Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:39:57.8774839Z 2022-03-17T20:39:57.8775268Z ---------------------------------------------------------------------- 2022-03-17T20:39:57.8776601Z Ran 1 test in 0.002s 2022-03-17T20:39:57.8776828Z 2022-03-17T20:39:57.8776977Z OK (skipped=1) 2022-03-17T20:39:57.8777161Z 2022-03-17T20:39:57.8777332Z Generating XML reports... 2022-03-17T20:39:57.8815417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203957.xml 2022-03-17T20:40:00.3770733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:00.3783627Z 2022-03-17T20:40:00.3784026Z Running tests... 2022-03-17T20:40:00.3784651Z ---------------------------------------------------------------------- 2022-03-17T20:40:00.3798633Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:40:00.9900961Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119041 2022-03-17T20:40:00.9985503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119042 2022-03-17T20:40:00.9986979Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:40:02.1017549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:02.1066806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:02.1068660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:02.1121055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:02.1122883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:02.2080412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:02.4932107Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:02.5889725Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:14.7273037Z ok (14.348s) 2022-03-17T20:40:14.7273527Z 2022-03-17T20:40:14.7274738Z ---------------------------------------------------------------------- 2022-03-17T20:40:14.7275726Z Ran 1 test in 14.349s 2022-03-17T20:40:14.7276197Z 2022-03-17T20:40:14.7276450Z OK 2022-03-17T20:40:14.7276786Z 2022-03-17T20:40:14.7277648Z Generating XML reports... 2022-03-17T20:40:14.7346919Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204000.xml 2022-03-17T20:40:17.6708578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:17.6718180Z 2022-03-17T20:40:17.6718697Z Running tests... 2022-03-17T20:40:17.6719366Z ---------------------------------------------------------------------- 2022-03-17T20:40:17.6739515Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:40:18.2993735Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119260 2022-03-17T20:40:18.3038120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119261 2022-03-17T20:40:18.3039808Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:40:19.3726710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:19.4358126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:19.4360328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:19.4434853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:19.4436629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:19.5372119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:19.8214731Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:19.9191526Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:29.5229447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.5231023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:32.1317226Z ok (14.459s) 2022-03-17T20:40:32.1317777Z 2022-03-17T20:40:32.1318873Z ---------------------------------------------------------------------- 2022-03-17T20:40:32.1319662Z Ran 1 test in 14.460s 2022-03-17T20:40:32.1320040Z 2022-03-17T20:40:32.1320208Z OK 2022-03-17T20:40:32.1320499Z 2022-03-17T20:40:32.1320776Z Generating XML reports... 2022-03-17T20:40:32.1394162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204017.xml 2022-03-17T20:40:35.0400696Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:35.0411633Z 2022-03-17T20:40:35.0411981Z Running tests... 2022-03-17T20:40:35.0412507Z ---------------------------------------------------------------------- 2022-03-17T20:40:35.0433830Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:40:35.6849676Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119479 2022-03-17T20:40:35.6886328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119480 2022-03-17T20:40:35.6887635Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:40:36.7723748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:36.7724999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:36.7726811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:36.7728590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:36.7729868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:36.7730990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:37.1551837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:37.2064743Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:47.0040801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:47.0042651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:49.9163425Z ok (14.874s) 2022-03-17T20:40:49.9163870Z 2022-03-17T20:40:49.9164822Z ---------------------------------------------------------------------- 2022-03-17T20:40:49.9165628Z Ran 1 test in 14.875s 2022-03-17T20:40:49.9165990Z 2022-03-17T20:40:49.9166189Z OK 2022-03-17T20:40:49.9166458Z 2022-03-17T20:40:49.9167651Z Generating XML reports... 2022-03-17T20:40:49.9257156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204035.xml 2022-03-17T20:40:52.5371966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:52.5382326Z 2022-03-17T20:40:52.5382573Z Running tests... 2022-03-17T20:40:52.5383074Z ---------------------------------------------------------------------- 2022-03-17T20:40:52.5399937Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:40:53.1842666Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119698 2022-03-17T20:40:53.1892353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119699 2022-03-17T20:40:53.1893851Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:40:54.2678998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:54.3172526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:54.3173511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:54.3177203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:54.3182277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:54.4188970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:54.8051891Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:40:54.8583955Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:06.9191184Z ok (14.380s) 2022-03-17T20:41:06.9191779Z 2022-03-17T20:41:06.9193015Z ---------------------------------------------------------------------- 2022-03-17T20:41:06.9193997Z Ran 1 test in 14.380s 2022-03-17T20:41:06.9194461Z 2022-03-17T20:41:06.9194701Z OK 2022-03-17T20:41:06.9195225Z 2022-03-17T20:41:06.9195674Z Generating XML reports... 2022-03-17T20:41:06.9286953Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204052.xml 2022-03-17T20:41:09.5472038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:09.5483317Z 2022-03-17T20:41:09.5483608Z Running tests... 2022-03-17T20:41:09.5484123Z ---------------------------------------------------------------------- 2022-03-17T20:41:09.5500931Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:41:09.5501257Z 2022-03-17T20:41:09.5501869Z ---------------------------------------------------------------------- 2022-03-17T20:41:09.5502235Z Ran 1 test in 0.002s 2022-03-17T20:41:09.5502405Z 2022-03-17T20:41:09.5502521Z OK (skipped=1) 2022-03-17T20:41:09.5502682Z 2022-03-17T20:41:09.5502808Z Generating XML reports... 2022-03-17T20:41:09.5540920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204109.xml 2022-03-17T20:41:12.0933753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:12.0945102Z 2022-03-17T20:41:12.0945342Z Running tests... 2022-03-17T20:41:12.0945995Z ---------------------------------------------------------------------- 2022-03-17T20:41:12.0963610Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:41:12.0964855Z 2022-03-17T20:41:12.0965960Z ---------------------------------------------------------------------- 2022-03-17T20:41:12.0966928Z Ran 1 test in 0.002s 2022-03-17T20:41:12.0967366Z 2022-03-17T20:41:12.0968164Z OK (skipped=1) 2022-03-17T20:41:12.0968564Z 2022-03-17T20:41:12.0968845Z Generating XML reports... 2022-03-17T20:41:12.1002987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204112.xml 2022-03-17T20:41:15.0600471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:15.0611189Z 2022-03-17T20:41:15.0611472Z Running tests... 2022-03-17T20:41:15.0612120Z ---------------------------------------------------------------------- 2022-03-17T20:41:15.0627911Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:41:15.0628334Z 2022-03-17T20:41:15.0628775Z ---------------------------------------------------------------------- 2022-03-17T20:41:15.0629202Z Ran 1 test in 0.002s 2022-03-17T20:41:15.0629394Z 2022-03-17T20:41:15.0629497Z OK (skipped=1) 2022-03-17T20:41:15.0629656Z 2022-03-17T20:41:15.0629782Z Generating XML reports... 2022-03-17T20:41:15.0668653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204115.xml 2022-03-17T20:41:17.6003406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:17.6015267Z 2022-03-17T20:41:17.6015677Z Running tests... 2022-03-17T20:41:17.6016857Z ---------------------------------------------------------------------- 2022-03-17T20:41:17.6029794Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:41:17.6030915Z 2022-03-17T20:41:17.6031725Z ---------------------------------------------------------------------- 2022-03-17T20:41:17.6032510Z Ran 1 test in 0.002s 2022-03-17T20:41:17.6032862Z 2022-03-17T20:41:17.6033091Z OK (skipped=1) 2022-03-17T20:41:17.6033450Z 2022-03-17T20:41:17.6033725Z Generating XML reports... 2022-03-17T20:41:17.6069500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204117.xml 2022-03-17T20:41:20.0766053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:20.0776930Z 2022-03-17T20:41:20.0777231Z Running tests... 2022-03-17T20:41:20.0777763Z ---------------------------------------------------------------------- 2022-03-17T20:41:20.0793710Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:41:20.0794206Z 2022-03-17T20:41:20.0794589Z ---------------------------------------------------------------------- 2022-03-17T20:41:20.0794956Z Ran 1 test in 0.002s 2022-03-17T20:41:20.0795109Z 2022-03-17T20:41:20.0795215Z OK (skipped=1) 2022-03-17T20:41:20.0795374Z 2022-03-17T20:41:20.0795495Z Generating XML reports... 2022-03-17T20:41:20.0832672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204120.xml 2022-03-17T20:41:22.5637198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:22.5650402Z 2022-03-17T20:41:22.5650601Z Running tests... 2022-03-17T20:41:22.5651117Z ---------------------------------------------------------------------- 2022-03-17T20:41:22.5678214Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:41:22.5678656Z 2022-03-17T20:41:22.5679054Z ---------------------------------------------------------------------- 2022-03-17T20:41:22.5679404Z Ran 1 test in 0.003s 2022-03-17T20:41:22.5679583Z 2022-03-17T20:41:22.5679687Z OK (skipped=1) 2022-03-17T20:41:22.5679849Z 2022-03-17T20:41:22.5679971Z Generating XML reports... 2022-03-17T20:41:22.5717148Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204122.xml 2022-03-17T20:41:25.0819757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:25.0831119Z 2022-03-17T20:41:25.0832003Z Running tests... 2022-03-17T20:41:25.0832686Z ---------------------------------------------------------------------- 2022-03-17T20:41:25.0848918Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:41:25.0849262Z 2022-03-17T20:41:25.0849717Z ---------------------------------------------------------------------- 2022-03-17T20:41:25.0850156Z Ran 1 test in 0.002s 2022-03-17T20:41:25.0850318Z 2022-03-17T20:41:25.0850422Z OK (skipped=1) 2022-03-17T20:41:25.0850583Z 2022-03-17T20:41:25.0850713Z Generating XML reports... 2022-03-17T20:41:25.0887132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204125.xml 2022-03-17T20:41:27.5912853Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:27.5922775Z 2022-03-17T20:41:27.5923323Z Running tests... 2022-03-17T20:41:27.5924726Z ---------------------------------------------------------------------- 2022-03-17T20:41:28.1917492Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120386 2022-03-17T20:41:28.1978788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120387 2022-03-17T20:41:28.1980554Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:41:29.3178675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:29.3361042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:29.3362186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:29.3382681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:29.3384350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:29.4371198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:29.7270970Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:29.8136835Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:31.6067245Z ok (4.014s) 2022-03-17T20:41:31.6067676Z 2022-03-17T20:41:31.6068708Z ---------------------------------------------------------------------- 2022-03-17T20:41:31.6069496Z Ran 1 test in 4.014s 2022-03-17T20:41:31.6069864Z 2022-03-17T20:41:31.6070050Z OK 2022-03-17T20:41:31.6070350Z 2022-03-17T20:41:31.6070634Z Generating XML reports... 2022-03-17T20:41:31.6144109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204127.xml 2022-03-17T20:41:34.5512446Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:34.5523165Z 2022-03-17T20:41:34.5523640Z Running tests... 2022-03-17T20:41:34.5524807Z ---------------------------------------------------------------------- 2022-03-17T20:41:35.1511098Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120607 2022-03-17T20:41:35.1549457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120608 2022-03-17T20:41:35.1550841Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:41:36.2366331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:36.2982905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:36.2985307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:36.3081126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:36.3086345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:36.3992318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:36.6911309Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:36.7813964Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:38.6684873Z ok (4.116s) 2022-03-17T20:41:38.6685300Z 2022-03-17T20:41:38.6686446Z ---------------------------------------------------------------------- 2022-03-17T20:41:38.6687506Z Ran 1 test in 4.116s 2022-03-17T20:41:38.6687866Z 2022-03-17T20:41:38.6688048Z OK 2022-03-17T20:41:38.6688347Z 2022-03-17T20:41:38.6688605Z Generating XML reports... 2022-03-17T20:41:38.6769132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204134.xml 2022-03-17T20:41:41.3400944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:41.3412092Z 2022-03-17T20:41:41.3412244Z Running tests... 2022-03-17T20:41:41.3412957Z ---------------------------------------------------------------------- 2022-03-17T20:41:41.3429622Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:41:41.3430152Z 2022-03-17T20:41:41.3430523Z ---------------------------------------------------------------------- 2022-03-17T20:41:41.3430867Z Ran 1 test in 0.002s 2022-03-17T20:41:41.3431036Z 2022-03-17T20:41:41.3431140Z OK (skipped=1) 2022-03-17T20:41:41.3431305Z 2022-03-17T20:41:41.3431424Z Generating XML reports... 2022-03-17T20:41:41.3467024Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204141.xml 2022-03-17T20:41:44.3430493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:44.3443561Z 2022-03-17T20:41:44.3443690Z Running tests... 2022-03-17T20:41:44.3444223Z ---------------------------------------------------------------------- 2022-03-17T20:41:44.3456188Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:41:44.3456478Z 2022-03-17T20:41:44.3456834Z ---------------------------------------------------------------------- 2022-03-17T20:41:44.3457201Z Ran 1 test in 0.002s 2022-03-17T20:41:44.3457369Z 2022-03-17T20:41:44.3457465Z OK (skipped=1) 2022-03-17T20:41:44.3457630Z 2022-03-17T20:41:44.3457749Z Generating XML reports... 2022-03-17T20:41:44.3497565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204144.xml 2022-03-17T20:41:46.8520978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:46.8532556Z 2022-03-17T20:41:46.8532836Z Running tests... 2022-03-17T20:41:46.8533351Z ---------------------------------------------------------------------- 2022-03-17T20:41:47.4867316Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120962 2022-03-17T20:41:47.4932428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120963 2022-03-17T20:41:47.4935610Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:41:48.6206345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:48.6419374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:48.6421978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:48.6514463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:48.6525229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:48.7430268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:49.0395904Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:49.1278051Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:51.1040203Z ok (4.250s) 2022-03-17T20:41:51.1040728Z 2022-03-17T20:41:51.1041738Z ---------------------------------------------------------------------- 2022-03-17T20:41:51.1042741Z Ran 1 test in 4.251s 2022-03-17T20:41:51.1043109Z 2022-03-17T20:41:51.1043295Z OK 2022-03-17T20:41:51.1043584Z 2022-03-17T20:41:51.1043851Z Generating XML reports... 2022-03-17T20:41:51.1117378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204146.xml 2022-03-17T20:41:54.0845963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:54.0858647Z 2022-03-17T20:41:54.0858861Z Running tests... 2022-03-17T20:41:54.0859522Z ---------------------------------------------------------------------- 2022-03-17T20:41:54.6806713Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121183 2022-03-17T20:41:54.6844375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121184 2022-03-17T20:41:54.6845914Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:41:55.7858437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:55.8185266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:55.8186249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:55.8263460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:55.8265559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:55.9196238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:56.2049009Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:56.3010907Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:41:58.2942263Z ok (4.208s) 2022-03-17T20:41:58.2942645Z 2022-03-17T20:41:58.2944366Z ---------------------------------------------------------------------- 2022-03-17T20:41:58.2945339Z Ran 1 test in 4.208s 2022-03-17T20:41:58.2945715Z 2022-03-17T20:41:58.2945903Z OK 2022-03-17T20:41:58.2946202Z 2022-03-17T20:41:58.2946487Z Generating XML reports... 2022-03-17T20:41:58.3011200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204154.xml 2022-03-17T20:42:00.9581350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:00.9592652Z 2022-03-17T20:42:00.9593031Z Running tests... 2022-03-17T20:42:00.9593560Z ---------------------------------------------------------------------- 2022-03-17T20:42:01.5583038Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121404 2022-03-17T20:42:01.5608154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121405 2022-03-17T20:42:01.5609970Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:42:02.6323119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:02.6601431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:02.6602471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:02.6625508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:02.6627456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:02.7608965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:03.3938827Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:42:03.4446243Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:42:05.1701102Z ok (4.210s) 2022-03-17T20:42:05.1701471Z 2022-03-17T20:42:05.1702515Z ---------------------------------------------------------------------- 2022-03-17T20:42:05.1703481Z Ran 1 test in 4.211s 2022-03-17T20:42:05.1703928Z 2022-03-17T20:42:05.1704167Z OK 2022-03-17T20:42:05.1704531Z 2022-03-17T20:42:05.1704871Z Generating XML reports... 2022-03-17T20:42:05.1783364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204200.xml 2022-03-17T20:42:07.7601260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:07.7613660Z 2022-03-17T20:42:07.7614256Z Running tests... 2022-03-17T20:42:07.7615696Z ---------------------------------------------------------------------- 2022-03-17T20:42:08.3598743Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121619 2022-03-17T20:42:08.3631181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121620 2022-03-17T20:42:08.3632768Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:42:09.4842318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:09.4850936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:09.4853252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:09.4942935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:09.4945951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:09.5866370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:09.6072190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:42:09.6073684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:42:09.6075762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:42:09.6175254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:42:10.3451322Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:42:10.3992181Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:42:10.9848354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:42:10.9851157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:42:10.9852533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:42:10.9854332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:42:11.0256503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:42:11.0257638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:42:11.0258232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:42:11.0258997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:42:12.6750707Z ok (4.913s) 2022-03-17T20:42:12.6751297Z 2022-03-17T20:42:12.6752379Z ---------------------------------------------------------------------- 2022-03-17T20:42:12.6753346Z Ran 1 test in 4.913s 2022-03-17T20:42:12.6753797Z 2022-03-17T20:42:12.6754034Z OK 2022-03-17T20:42:12.6754376Z 2022-03-17T20:42:12.6754663Z Generating XML reports... 2022-03-17T20:42:12.6821066Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204207.xml 2022-03-17T20:42:15.2047538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:15.2060634Z 2022-03-17T20:42:15.2060843Z Running tests... 2022-03-17T20:42:15.2061379Z ---------------------------------------------------------------------- 2022-03-17T20:42:15.2076557Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:15.2076918Z 2022-03-17T20:42:15.2077429Z ---------------------------------------------------------------------- 2022-03-17T20:42:15.2077798Z Ran 1 test in 0.002s 2022-03-17T20:42:15.2077977Z 2022-03-17T20:42:15.2078086Z OK (skipped=1) 2022-03-17T20:42:15.2078250Z 2022-03-17T20:42:15.2078366Z Generating XML reports... 2022-03-17T20:42:15.2113915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204215.xml 2022-03-17T20:42:17.6690523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:17.6701443Z 2022-03-17T20:42:17.6702062Z Running tests... 2022-03-17T20:42:17.6703268Z ---------------------------------------------------------------------- 2022-03-17T20:42:17.6718646Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:17.6719515Z 2022-03-17T20:42:17.6720495Z ---------------------------------------------------------------------- 2022-03-17T20:42:17.6721474Z Ran 1 test in 0.002s 2022-03-17T20:42:17.6722151Z 2022-03-17T20:42:17.6722439Z OK (skipped=1) 2022-03-17T20:42:17.6722898Z 2022-03-17T20:42:17.6723176Z Generating XML reports... 2022-03-17T20:42:17.6757612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204217.xml 2022-03-17T20:42:20.1431432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:20.1444526Z 2022-03-17T20:42:20.1444766Z Running tests... 2022-03-17T20:42:20.1445311Z ---------------------------------------------------------------------- 2022-03-17T20:42:20.1458369Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:20.1458680Z 2022-03-17T20:42:20.1459042Z ---------------------------------------------------------------------- 2022-03-17T20:42:20.1459394Z Ran 1 test in 0.002s 2022-03-17T20:42:20.1459560Z 2022-03-17T20:42:20.1459653Z OK (skipped=1) 2022-03-17T20:42:20.1459812Z 2022-03-17T20:42:20.1459932Z Generating XML reports... 2022-03-17T20:42:20.1498094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204220.xml 2022-03-17T20:42:22.6170360Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:22.6183194Z 2022-03-17T20:42:22.6183325Z Running tests... 2022-03-17T20:42:22.6183855Z ---------------------------------------------------------------------- 2022-03-17T20:42:22.6197401Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:22.6197899Z 2022-03-17T20:42:22.6198355Z ---------------------------------------------------------------------- 2022-03-17T20:42:22.6198774Z Ran 1 test in 0.002s 2022-03-17T20:42:22.6198949Z 2022-03-17T20:42:22.6199043Z OK (skipped=1) 2022-03-17T20:42:22.6199210Z 2022-03-17T20:42:22.6200149Z Generating XML reports... 2022-03-17T20:42:22.6238736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204222.xml 2022-03-17T20:42:25.0995163Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:25.1008839Z 2022-03-17T20:42:25.1009127Z Running tests... 2022-03-17T20:42:25.1009657Z ---------------------------------------------------------------------- 2022-03-17T20:42:25.1022827Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:25.1023189Z 2022-03-17T20:42:25.1023757Z ---------------------------------------------------------------------- 2022-03-17T20:42:25.1024104Z Ran 1 test in 0.002s 2022-03-17T20:42:25.1024271Z 2022-03-17T20:42:25.1024380Z OK (skipped=1) 2022-03-17T20:42:25.1024541Z 2022-03-17T20:42:25.1024665Z Generating XML reports... 2022-03-17T20:42:25.1062482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204225.xml 2022-03-17T20:42:27.6112780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:27.6125740Z 2022-03-17T20:42:27.6125941Z Running tests... 2022-03-17T20:42:27.6126590Z ---------------------------------------------------------------------- 2022-03-17T20:42:27.6146136Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:27.6146550Z 2022-03-17T20:42:27.6146997Z ---------------------------------------------------------------------- 2022-03-17T20:42:27.6147431Z Ran 1 test in 0.002s 2022-03-17T20:42:27.6147636Z 2022-03-17T20:42:27.6147731Z OK (skipped=1) 2022-03-17T20:42:27.6147886Z 2022-03-17T20:42:27.6148006Z Generating XML reports... 2022-03-17T20:42:27.6183310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204227.xml 2022-03-17T20:42:30.3185066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:30.3197693Z 2022-03-17T20:42:30.3198010Z Running tests... 2022-03-17T20:42:30.3198539Z ---------------------------------------------------------------------- 2022-03-17T20:42:30.3212813Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:30.3213275Z 2022-03-17T20:42:30.3213726Z ---------------------------------------------------------------------- 2022-03-17T20:42:30.3214088Z Ran 1 test in 0.002s 2022-03-17T20:42:30.3214248Z 2022-03-17T20:42:30.3214350Z OK (skipped=1) 2022-03-17T20:42:30.3214516Z 2022-03-17T20:42:30.3214637Z Generating XML reports... 2022-03-17T20:42:30.3252085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204230.xml 2022-03-17T20:42:32.9242813Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:32.9254286Z 2022-03-17T20:42:32.9254887Z Running tests... 2022-03-17T20:42:32.9256041Z ---------------------------------------------------------------------- 2022-03-17T20:42:32.9271490Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:32.9272408Z 2022-03-17T20:42:32.9273428Z ---------------------------------------------------------------------- 2022-03-17T20:42:32.9274368Z Ran 1 test in 0.002s 2022-03-17T20:42:32.9274843Z 2022-03-17T20:42:32.9275082Z OK (skipped=1) 2022-03-17T20:42:32.9275436Z 2022-03-17T20:42:32.9275697Z Generating XML reports... 2022-03-17T20:42:32.9311717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204232.xml 2022-03-17T20:42:35.4126280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:35.4138235Z 2022-03-17T20:42:35.4138387Z Running tests... 2022-03-17T20:42:35.4139168Z ---------------------------------------------------------------------- 2022-03-17T20:42:35.4157783Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:35.4158118Z 2022-03-17T20:42:35.4158542Z ---------------------------------------------------------------------- 2022-03-17T20:42:35.4158887Z Ran 1 test in 0.002s 2022-03-17T20:42:35.4159064Z 2022-03-17T20:42:35.4159183Z OK (skipped=1) 2022-03-17T20:42:35.4159347Z 2022-03-17T20:42:35.4159464Z Generating XML reports... 2022-03-17T20:42:35.4195740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204235.xml 2022-03-17T20:42:37.8908536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:37.8919686Z 2022-03-17T20:42:37.8919822Z Running tests... 2022-03-17T20:42:37.8920368Z ---------------------------------------------------------------------- 2022-03-17T20:42:37.8937497Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:37.8939534Z 2022-03-17T20:42:37.8939910Z ---------------------------------------------------------------------- 2022-03-17T20:42:37.8940283Z Ran 1 test in 0.002s 2022-03-17T20:42:37.8940464Z 2022-03-17T20:42:37.8940568Z OK (skipped=1) 2022-03-17T20:42:37.8940723Z 2022-03-17T20:42:37.8940851Z Generating XML reports... 2022-03-17T20:42:37.8975662Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204237.xml 2022-03-17T20:42:40.4042344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:40.4053568Z 2022-03-17T20:42:40.4053878Z Running tests... 2022-03-17T20:42:40.4054384Z ---------------------------------------------------------------------- 2022-03-17T20:42:40.4072305Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:40.4072771Z 2022-03-17T20:42:40.4073211Z ---------------------------------------------------------------------- 2022-03-17T20:42:40.4073635Z Ran 1 test in 0.002s 2022-03-17T20:42:40.4073840Z 2022-03-17T20:42:40.4074003Z OK (skipped=1) 2022-03-17T20:42:40.4074166Z 2022-03-17T20:42:40.4074277Z Generating XML reports... 2022-03-17T20:42:40.4112793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204240.xml 2022-03-17T20:42:42.8887936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:42.8901674Z 2022-03-17T20:42:42.8902112Z Running tests... 2022-03-17T20:42:42.8903261Z ---------------------------------------------------------------------- 2022-03-17T20:42:42.8916987Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:42.8917790Z 2022-03-17T20:42:42.8918691Z ---------------------------------------------------------------------- 2022-03-17T20:42:42.8919471Z Ran 1 test in 0.002s 2022-03-17T20:42:42.8919865Z 2022-03-17T20:42:42.8920102Z OK (skipped=1) 2022-03-17T20:42:42.8920455Z 2022-03-17T20:42:42.8921126Z Generating XML reports... 2022-03-17T20:42:42.8957054Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204242.xml 2022-03-17T20:42:45.4151901Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:45.4159889Z 2022-03-17T20:42:45.4160292Z Running tests... 2022-03-17T20:42:45.4161552Z ---------------------------------------------------------------------- 2022-03-17T20:42:45.4179615Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:45.4180673Z 2022-03-17T20:42:45.4181481Z ---------------------------------------------------------------------- 2022-03-17T20:42:45.4182539Z Ran 1 test in 0.002s 2022-03-17T20:42:45.4182956Z 2022-03-17T20:42:45.4183229Z OK (skipped=1) 2022-03-17T20:42:45.4183702Z 2022-03-17T20:42:45.4184078Z Generating XML reports... 2022-03-17T20:42:45.4218411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204245.xml 2022-03-17T20:42:47.8828776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:47.8838381Z 2022-03-17T20:42:47.8838714Z Running tests... 2022-03-17T20:42:47.8839232Z ---------------------------------------------------------------------- 2022-03-17T20:42:47.8857720Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:47.8858050Z 2022-03-17T20:42:47.8858730Z ---------------------------------------------------------------------- 2022-03-17T20:42:47.8859267Z Ran 1 test in 0.002s 2022-03-17T20:42:47.8859438Z 2022-03-17T20:42:47.8859542Z OK (skipped=1) 2022-03-17T20:42:47.8859706Z 2022-03-17T20:42:47.8859830Z Generating XML reports... 2022-03-17T20:42:47.8899122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204247.xml 2022-03-17T20:42:50.4082157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:50.4096401Z 2022-03-17T20:42:50.4096918Z Running tests... 2022-03-17T20:42:50.4097444Z ---------------------------------------------------------------------- 2022-03-17T20:42:50.4117670Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:50.4117986Z 2022-03-17T20:42:50.4118410Z ---------------------------------------------------------------------- 2022-03-17T20:42:50.4118778Z Ran 1 test in 0.002s 2022-03-17T20:42:50.4118951Z 2022-03-17T20:42:50.4119071Z OK (skipped=1) 2022-03-17T20:42:50.4119237Z 2022-03-17T20:42:50.4119350Z Generating XML reports... 2022-03-17T20:42:50.4157191Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204250.xml 2022-03-17T20:42:53.1189417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:53.1200127Z 2022-03-17T20:42:53.1200401Z Running tests... 2022-03-17T20:42:53.1200926Z ---------------------------------------------------------------------- 2022-03-17T20:42:53.1217147Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:53.1217458Z 2022-03-17T20:42:53.1218044Z ---------------------------------------------------------------------- 2022-03-17T20:42:53.1218411Z Ran 1 test in 0.002s 2022-03-17T20:42:53.1218585Z 2022-03-17T20:42:53.1218704Z OK (skipped=1) 2022-03-17T20:42:53.1218875Z 2022-03-17T20:42:53.1219008Z Generating XML reports... 2022-03-17T20:42:53.1255713Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204253.xml 2022-03-17T20:42:55.6376562Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:55.6392856Z 2022-03-17T20:42:55.6393191Z Running tests... 2022-03-17T20:42:55.6394370Z ---------------------------------------------------------------------- 2022-03-17T20:42:55.6413863Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:55.6414561Z 2022-03-17T20:42:55.6415399Z ---------------------------------------------------------------------- 2022-03-17T20:42:55.6416172Z Ran 1 test in 0.002s 2022-03-17T20:42:55.6416542Z 2022-03-17T20:42:55.6416779Z OK (skipped=1) 2022-03-17T20:42:55.6417144Z 2022-03-17T20:42:55.6417426Z Generating XML reports... 2022-03-17T20:42:55.6455436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204255.xml 2022-03-17T20:42:58.3291578Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:58.3306018Z 2022-03-17T20:42:58.3306376Z Running tests... 2022-03-17T20:42:58.3306987Z ---------------------------------------------------------------------- 2022-03-17T20:42:58.3323900Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:42:58.3324265Z 2022-03-17T20:42:58.3324777Z ---------------------------------------------------------------------- 2022-03-17T20:42:58.3325104Z Ran 1 test in 0.002s 2022-03-17T20:42:58.3325269Z 2022-03-17T20:42:58.3325369Z OK (skipped=1) 2022-03-17T20:42:58.3325527Z 2022-03-17T20:42:58.3325659Z Generating XML reports... 2022-03-17T20:42:58.3364696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204258.xml 2022-03-17T20:43:00.8365889Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:00.8379973Z 2022-03-17T20:43:00.8380348Z Running tests... 2022-03-17T20:43:00.8381002Z ---------------------------------------------------------------------- 2022-03-17T20:43:00.8395244Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:00.8395632Z 2022-03-17T20:43:00.8396066Z ---------------------------------------------------------------------- 2022-03-17T20:43:00.8396499Z Ran 1 test in 0.002s 2022-03-17T20:43:00.8396706Z 2022-03-17T20:43:00.8396831Z OK (skipped=1) 2022-03-17T20:43:00.8396983Z 2022-03-17T20:43:00.8397104Z Generating XML reports... 2022-03-17T20:43:00.8434502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204300.xml 2022-03-17T20:43:03.8566306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:03.8578109Z 2022-03-17T20:43:03.8578666Z Running tests... 2022-03-17T20:43:03.8579225Z ---------------------------------------------------------------------- 2022-03-17T20:43:03.8595468Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:03.8595775Z 2022-03-17T20:43:03.8596118Z ---------------------------------------------------------------------- 2022-03-17T20:43:03.8596544Z Ran 1 test in 0.002s 2022-03-17T20:43:03.8596712Z 2022-03-17T20:43:03.8596828Z OK (skipped=1) 2022-03-17T20:43:03.8596979Z 2022-03-17T20:43:03.8597103Z Generating XML reports... 2022-03-17T20:43:03.8632945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204303.xml 2022-03-17T20:43:06.4604703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:06.4618298Z 2022-03-17T20:43:06.4618790Z Running tests... 2022-03-17T20:43:06.4619941Z ---------------------------------------------------------------------- 2022-03-17T20:43:06.4634817Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:06.4635691Z 2022-03-17T20:43:06.4636474Z ---------------------------------------------------------------------- 2022-03-17T20:43:06.4637236Z Ran 1 test in 0.002s 2022-03-17T20:43:06.4637603Z 2022-03-17T20:43:06.4637834Z OK (skipped=1) 2022-03-17T20:43:06.4638175Z 2022-03-17T20:43:06.4638443Z Generating XML reports... 2022-03-17T20:43:06.4676799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204306.xml 2022-03-17T20:43:08.9407398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:08.9419138Z 2022-03-17T20:43:08.9419492Z Running tests... 2022-03-17T20:43:08.9420027Z ---------------------------------------------------------------------- 2022-03-17T20:43:08.9438710Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:08.9440448Z 2022-03-17T20:43:08.9441603Z ---------------------------------------------------------------------- 2022-03-17T20:43:08.9443095Z Ran 1 test in 0.002s 2022-03-17T20:43:08.9443678Z 2022-03-17T20:43:08.9444303Z OK (skipped=1) 2022-03-17T20:43:08.9445032Z 2022-03-17T20:43:08.9445815Z Generating XML reports... 2022-03-17T20:43:08.9477545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204308.xml 2022-03-17T20:43:11.5542458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:11.5554121Z 2022-03-17T20:43:11.5554551Z Running tests... 2022-03-17T20:43:11.5555083Z ---------------------------------------------------------------------- 2022-03-17T20:43:11.5573010Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:11.5573415Z 2022-03-17T20:43:11.5573857Z ---------------------------------------------------------------------- 2022-03-17T20:43:11.5574711Z Ran 1 test in 0.002s 2022-03-17T20:43:11.5574882Z 2022-03-17T20:43:11.5574985Z OK (skipped=1) 2022-03-17T20:43:11.5575138Z 2022-03-17T20:43:11.5575263Z Generating XML reports... 2022-03-17T20:43:11.5620578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204311.xml 2022-03-17T20:43:14.1364526Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:14.1379860Z 2022-03-17T20:43:14.1380210Z Running tests... 2022-03-17T20:43:14.1380744Z ---------------------------------------------------------------------- 2022-03-17T20:43:14.1397700Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:14.1398339Z 2022-03-17T20:43:14.1399263Z ---------------------------------------------------------------------- 2022-03-17T20:43:14.1400045Z Ran 1 test in 0.002s 2022-03-17T20:43:14.1400426Z 2022-03-17T20:43:14.1400684Z OK (skipped=1) 2022-03-17T20:43:14.1401043Z 2022-03-17T20:43:14.1401349Z Generating XML reports... 2022-03-17T20:43:14.1437779Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204314.xml 2022-03-17T20:43:16.8988519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:16.9003407Z 2022-03-17T20:43:16.9003535Z Running tests... 2022-03-17T20:43:16.9004103Z ---------------------------------------------------------------------- 2022-03-17T20:43:16.9026641Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:16.9026954Z 2022-03-17T20:43:16.9027336Z ---------------------------------------------------------------------- 2022-03-17T20:43:16.9027701Z Ran 1 test in 0.002s 2022-03-17T20:43:16.9027875Z 2022-03-17T20:43:16.9027984Z OK (skipped=1) 2022-03-17T20:43:16.9028134Z 2022-03-17T20:43:16.9028268Z Generating XML reports... 2022-03-17T20:43:16.9067306Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204316.xml 2022-03-17T20:43:19.4586653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:19.4598357Z 2022-03-17T20:43:19.4598894Z Running tests... 2022-03-17T20:43:19.4599705Z ---------------------------------------------------------------------- 2022-03-17T20:43:19.4618176Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:19.4618583Z 2022-03-17T20:43:19.4619061Z ---------------------------------------------------------------------- 2022-03-17T20:43:19.4619500Z Ran 1 test in 0.002s 2022-03-17T20:43:19.4619705Z 2022-03-17T20:43:19.4619832Z OK (skipped=1) 2022-03-17T20:43:19.4619985Z 2022-03-17T20:43:19.4620107Z Generating XML reports... 2022-03-17T20:43:19.4660153Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204319.xml 2022-03-17T20:43:21.9331839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:21.9345314Z 2022-03-17T20:43:21.9345716Z Running tests... 2022-03-17T20:43:21.9346261Z ---------------------------------------------------------------------- 2022-03-17T20:43:21.9365054Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:21.9365465Z 2022-03-17T20:43:21.9365923Z ---------------------------------------------------------------------- 2022-03-17T20:43:21.9366364Z Ran 1 test in 0.002s 2022-03-17T20:43:21.9366558Z 2022-03-17T20:43:21.9366705Z OK (skipped=1) 2022-03-17T20:43:21.9366998Z 2022-03-17T20:43:21.9367166Z Generating XML reports... 2022-03-17T20:43:21.9403499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204321.xml 2022-03-17T20:43:24.4326984Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:24.4341561Z 2022-03-17T20:43:24.4341917Z Running tests... 2022-03-17T20:43:24.4342472Z ---------------------------------------------------------------------- 2022-03-17T20:43:25.0487231Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123682 2022-03-17T20:43:25.0528593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123683 2022-03-17T20:43:25.0529957Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:43:26.1245524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:26.1838468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:26.1839483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:26.1851086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:26.1859524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:26.2853314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:26.6651593Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:26.7158802Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:28.5622605Z ok (4.127s) 2022-03-17T20:43:28.5623014Z 2022-03-17T20:43:28.5624032Z ---------------------------------------------------------------------- 2022-03-17T20:43:28.5624812Z Ran 1 test in 4.128s 2022-03-17T20:43:28.5625177Z 2022-03-17T20:43:28.5625361Z OK 2022-03-17T20:43:28.5625666Z 2022-03-17T20:43:28.5625954Z Generating XML reports... 2022-03-17T20:43:28.5702764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204324.xml 2022-03-17T20:43:31.2445182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:31.2458414Z 2022-03-17T20:43:31.2458744Z Running tests... 2022-03-17T20:43:31.2459901Z ---------------------------------------------------------------------- 2022-03-17T20:43:31.2479745Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:31.2480096Z 2022-03-17T20:43:31.2480580Z ---------------------------------------------------------------------- 2022-03-17T20:43:31.2480941Z Ran 1 test in 0.002s 2022-03-17T20:43:31.2481106Z 2022-03-17T20:43:31.2481205Z OK (skipped=1) 2022-03-17T20:43:31.2481362Z 2022-03-17T20:43:31.2481486Z Generating XML reports... 2022-03-17T20:43:31.2522461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204331.xml 2022-03-17T20:43:33.8159933Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:33.8174698Z 2022-03-17T20:43:33.8174974Z Running tests... 2022-03-17T20:43:33.8175556Z ---------------------------------------------------------------------- 2022-03-17T20:43:33.8197980Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:33.8198289Z 2022-03-17T20:43:33.8198710Z ---------------------------------------------------------------------- 2022-03-17T20:43:33.8199071Z Ran 1 test in 0.002s 2022-03-17T20:43:33.8199229Z 2022-03-17T20:43:33.8199341Z OK (skipped=1) 2022-03-17T20:43:33.8199503Z 2022-03-17T20:43:33.8199633Z Generating XML reports... 2022-03-17T20:43:33.8238315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204333.xml 2022-03-17T20:43:36.5725330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:36.5740436Z 2022-03-17T20:43:36.5740739Z Running tests... 2022-03-17T20:43:36.5741272Z ---------------------------------------------------------------------- 2022-03-17T20:43:36.5761533Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:36.5762067Z 2022-03-17T20:43:36.5762508Z ---------------------------------------------------------------------- 2022-03-17T20:43:36.5762953Z Ran 1 test in 0.002s 2022-03-17T20:43:36.5763240Z 2022-03-17T20:43:36.5763435Z OK (skipped=1) 2022-03-17T20:43:36.5763589Z 2022-03-17T20:43:36.5763712Z Generating XML reports... 2022-03-17T20:43:36.5801771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204336.xml 2022-03-17T20:43:39.1627390Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:39.1640565Z 2022-03-17T20:43:39.1640856Z Running tests... 2022-03-17T20:43:39.1641522Z ---------------------------------------------------------------------- 2022-03-17T20:43:39.7621124Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124098 2022-03-17T20:43:39.7649575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124099 2022-03-17T20:43:39.7651206Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:43:40.8857033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:40.8889998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:40.8891021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:40.8955700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:40.8958539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:40.9904517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:41.3727111Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:41.4299378Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:43.4751613Z ok (4.310s) 2022-03-17T20:43:43.4752141Z 2022-03-17T20:43:43.4753350Z ---------------------------------------------------------------------- 2022-03-17T20:43:43.4754275Z Ran 1 test in 4.311s 2022-03-17T20:43:43.4754743Z 2022-03-17T20:43:43.4754927Z OK 2022-03-17T20:43:43.4755218Z 2022-03-17T20:43:43.4755502Z Generating XML reports... 2022-03-17T20:43:43.4829062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204339.xml 2022-03-17T20:43:46.2042613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:46.2060194Z 2022-03-17T20:43:46.2060590Z Running tests... 2022-03-17T20:43:46.2061255Z ---------------------------------------------------------------------- 2022-03-17T20:43:46.8049623Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124319 2022-03-17T20:43:46.8087175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124320 2022-03-17T20:43:46.8088575Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:43:47.8667542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:47.9419822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:47.9420958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:47.9474910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:47.9477180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:48.0429298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:48.4208505Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:48.4670704Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:50.1192272Z ok (3.913s) 2022-03-17T20:43:50.1192705Z 2022-03-17T20:43:50.1193969Z ---------------------------------------------------------------------- 2022-03-17T20:43:50.1194991Z Ran 1 test in 3.913s 2022-03-17T20:43:50.1195376Z 2022-03-17T20:43:50.1195588Z OK 2022-03-17T20:43:50.1195922Z 2022-03-17T20:43:50.1196209Z Generating XML reports... 2022-03-17T20:43:50.1266205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204346.xml 2022-03-17T20:43:52.6697985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:52.6713742Z 2022-03-17T20:43:52.6714742Z Running tests... 2022-03-17T20:43:52.6715969Z ---------------------------------------------------------------------- 2022-03-17T20:43:53.3047226Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124540 2022-03-17T20:43:53.3086500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124541 2022-03-17T20:43:53.3087905Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:43:54.4191610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:54.4371123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:54.4373469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:54.4393073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:54.4395289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:54.5380268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:54.9149141Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:54.9604337Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:43:56.7186878Z ok (4.047s) 2022-03-17T20:43:56.7187320Z 2022-03-17T20:43:56.7188985Z ---------------------------------------------------------------------- 2022-03-17T20:43:56.7189969Z Ran 1 test in 4.047s 2022-03-17T20:43:56.7190430Z 2022-03-17T20:43:56.7190666Z OK 2022-03-17T20:43:56.7191072Z 2022-03-17T20:43:56.7191380Z Generating XML reports... 2022-03-17T20:43:56.7270646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204352.xml 2022-03-17T20:43:59.3877472Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:59.3895595Z 2022-03-17T20:43:59.3896077Z Running tests... 2022-03-17T20:43:59.3896742Z ---------------------------------------------------------------------- 2022-03-17T20:43:59.3912738Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:43:59.3913193Z 2022-03-17T20:43:59.3913637Z ---------------------------------------------------------------------- 2022-03-17T20:43:59.3914071Z Ran 1 test in 0.002s 2022-03-17T20:43:59.3914628Z 2022-03-17T20:43:59.3914733Z OK (skipped=1) 2022-03-17T20:43:59.3914893Z 2022-03-17T20:43:59.3915017Z Generating XML reports... 2022-03-17T20:43:59.3951552Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204359.xml 2022-03-17T20:44:01.8517910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:01.8533408Z 2022-03-17T20:44:01.8533824Z Running tests... 2022-03-17T20:44:01.8534370Z ---------------------------------------------------------------------- 2022-03-17T20:44:01.8549378Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:44:01.8549672Z 2022-03-17T20:44:01.8550027Z ---------------------------------------------------------------------- 2022-03-17T20:44:01.8550382Z Ran 1 test in 0.002s 2022-03-17T20:44:01.8550551Z 2022-03-17T20:44:01.8550646Z OK (skipped=1) 2022-03-17T20:44:01.8550808Z 2022-03-17T20:44:01.8552068Z Generating XML reports... 2022-03-17T20:44:01.8592085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204401.xml 2022-03-17T20:44:04.3582342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:04.3595191Z 2022-03-17T20:44:04.3595718Z Running tests... 2022-03-17T20:44:04.3596894Z ---------------------------------------------------------------------- 2022-03-17T20:44:04.9876920Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124895 2022-03-17T20:44:04.9923952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124896 2022-03-17T20:44:04.9925328Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:06.0978684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:06.1301269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:06.1303582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:06.1386953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:06.1397590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:06.2317560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:08.5025190Z skip (4.143s) 2022-03-17T20:44:08.5025697Z 2022-03-17T20:44:08.5027297Z ---------------------------------------------------------------------- 2022-03-17T20:44:08.5028240Z Ran 1 test in 4.143s 2022-03-17T20:44:08.5028692Z 2022-03-17T20:44:08.5029012Z OK (skipped=1) 2022-03-17T20:44:08.5029466Z 2022-03-17T20:44:08.5029783Z Generating XML reports... 2022-03-17T20:44:08.5105008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204404.xml 2022-03-17T20:44:11.2343253Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:11.2354793Z 2022-03-17T20:44:11.2355421Z Running tests... 2022-03-17T20:44:11.2356883Z ---------------------------------------------------------------------- 2022-03-17T20:44:11.8357543Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125103 2022-03-17T20:44:11.8388781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125104 2022-03-17T20:44:11.8390403Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:12.9410777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:12.9741787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:12.9743877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:12.9815521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:12.9817190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:13.0757886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:14.8469690Z skip (3.611s) 2022-03-17T20:44:14.8470817Z 2022-03-17T20:44:14.8472328Z ---------------------------------------------------------------------- 2022-03-17T20:44:14.8473186Z Ran 1 test in 3.611s 2022-03-17T20:44:14.8473586Z 2022-03-17T20:44:14.8473886Z OK (skipped=1) 2022-03-17T20:44:14.8474241Z 2022-03-17T20:44:14.8474526Z Generating XML reports... 2022-03-17T20:44:14.8543233Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204411.xml 2022-03-17T20:44:17.7239738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:17.7248266Z 2022-03-17T20:44:17.7248799Z Running tests... 2022-03-17T20:44:17.7250124Z ---------------------------------------------------------------------- 2022-03-17T20:44:17.7269937Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:44:17.7270883Z 2022-03-17T20:44:17.7271668Z ---------------------------------------------------------------------- 2022-03-17T20:44:17.7272417Z Ran 1 test in 0.002s 2022-03-17T20:44:17.7272785Z 2022-03-17T20:44:17.7273027Z OK (skipped=1) 2022-03-17T20:44:17.7273377Z 2022-03-17T20:44:17.7273650Z Generating XML reports... 2022-03-17T20:44:17.7309426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204417.xml 2022-03-17T20:44:20.6294381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:20.6306349Z 2022-03-17T20:44:20.6306574Z Running tests... 2022-03-17T20:44:20.6307029Z ---------------------------------------------------------------------- 2022-03-17T20:44:21.2437347Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125378 2022-03-17T20:44:21.2473840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125379 2022-03-17T20:44:21.2475207Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:22.3200347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:22.3562588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:22.3565019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:22.3608134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:22.3609758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:22.4571713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:24.2574374Z skip (3.626s) 2022-03-17T20:44:24.2574790Z 2022-03-17T20:44:24.2576775Z ---------------------------------------------------------------------- 2022-03-17T20:44:24.2577757Z Ran 1 test in 3.626s 2022-03-17T20:44:24.2578134Z 2022-03-17T20:44:24.2578383Z OK (skipped=1) 2022-03-17T20:44:24.2578736Z 2022-03-17T20:44:24.2579705Z Generating XML reports... 2022-03-17T20:44:24.2659039Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204420.xml 2022-03-17T20:44:26.8854091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:26.8866634Z 2022-03-17T20:44:26.8867282Z Running tests... 2022-03-17T20:44:26.8868680Z ---------------------------------------------------------------------- 2022-03-17T20:44:26.8886395Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:44:26.8887171Z 2022-03-17T20:44:26.8888149Z ---------------------------------------------------------------------- 2022-03-17T20:44:26.8889069Z Ran 1 test in 0.002s 2022-03-17T20:44:26.8889513Z 2022-03-17T20:44:26.8889746Z OK (skipped=1) 2022-03-17T20:44:26.8890091Z 2022-03-17T20:44:26.8890363Z Generating XML reports... 2022-03-17T20:44:26.8927274Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204426.xml 2022-03-17T20:44:29.4243940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:29.4258111Z 2022-03-17T20:44:29.4258622Z Running tests... 2022-03-17T20:44:29.4259754Z ---------------------------------------------------------------------- 2022-03-17T20:44:30.0207714Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125653 2022-03-17T20:44:30.0241726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125654 2022-03-17T20:44:30.0243507Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:31.1281434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:31.1487773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:31.1489796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:31.1587012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:31.1597025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:31.2496669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:33.0330355Z skip (3.607s) 2022-03-17T20:44:33.0330816Z 2022-03-17T20:44:33.0331980Z ---------------------------------------------------------------------- 2022-03-17T20:44:33.0332944Z Ran 1 test in 3.607s 2022-03-17T20:44:33.0333412Z 2022-03-17T20:44:33.0333698Z OK (skipped=1) 2022-03-17T20:44:33.0334116Z 2022-03-17T20:44:33.0336245Z Generating XML reports... 2022-03-17T20:44:33.0416479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204429.xml 2022-03-17T20:44:35.6621314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:35.6633713Z 2022-03-17T20:44:35.6633992Z Running tests... 2022-03-17T20:44:35.6634511Z ---------------------------------------------------------------------- 2022-03-17T20:44:35.6650195Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:44:35.6650542Z 2022-03-17T20:44:35.6651007Z ---------------------------------------------------------------------- 2022-03-17T20:44:35.6651454Z Ran 1 test in 0.002s 2022-03-17T20:44:35.6651666Z 2022-03-17T20:44:35.6651787Z OK (skipped=1) 2022-03-17T20:44:35.6651992Z 2022-03-17T20:44:35.6652146Z Generating XML reports... 2022-03-17T20:44:35.6691352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204435.xml 2022-03-17T20:44:38.1326676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:38.1339257Z 2022-03-17T20:44:38.1339533Z Running tests... 2022-03-17T20:44:38.1340186Z ---------------------------------------------------------------------- 2022-03-17T20:44:38.1358223Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:44:38.1358663Z 2022-03-17T20:44:38.1359120Z ---------------------------------------------------------------------- 2022-03-17T20:44:38.1359543Z Ran 1 test in 0.002s 2022-03-17T20:44:38.1359751Z 2022-03-17T20:44:38.1359882Z OK (skipped=1) 2022-03-17T20:44:38.1360046Z 2022-03-17T20:44:38.1360168Z Generating XML reports... 2022-03-17T20:44:38.1398105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204438.xml 2022-03-17T20:44:40.9940240Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:40.9952825Z 2022-03-17T20:44:40.9953402Z Running tests... 2022-03-17T20:44:40.9954674Z ---------------------------------------------------------------------- 2022-03-17T20:44:41.5927465Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125995 2022-03-17T20:44:41.5956117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125996 2022-03-17T20:44:41.5957434Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:42.6973123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:42.7132338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:42.7133209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:42.7177510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:42.7188139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:42.8146233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:43.1039986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:44:43.2016650Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:44:45.5056500Z ok (4.510s) 2022-03-17T20:44:45.5056899Z 2022-03-17T20:44:45.5057805Z ---------------------------------------------------------------------- 2022-03-17T20:44:45.5058572Z Ran 1 test in 4.510s 2022-03-17T20:44:45.5058954Z 2022-03-17T20:44:45.5059170Z OK 2022-03-17T20:44:45.5059472Z 2022-03-17T20:44:45.5059765Z Generating XML reports... 2022-03-17T20:44:45.5140210Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204440.xml 2022-03-17T20:44:48.1281705Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:48.1297921Z 2022-03-17T20:44:48.1298653Z Running tests... 2022-03-17T20:44:48.1300067Z ---------------------------------------------------------------------- 2022-03-17T20:44:48.7406860Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126216 2022-03-17T20:44:48.7495867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126217 2022-03-17T20:44:48.7496988Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:49.8341214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:49.8730068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:49.8731122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:49.8750663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:49.8759051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:49.9741250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:50.3518700Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:44:50.4077241Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:44:52.2585852Z ok (4.128s) 2022-03-17T20:44:52.2586882Z 2022-03-17T20:44:52.2588475Z ---------------------------------------------------------------------- 2022-03-17T20:44:52.2589465Z Ran 1 test in 4.129s 2022-03-17T20:44:52.2589845Z 2022-03-17T20:44:52.2590058Z OK 2022-03-17T20:44:52.2590346Z 2022-03-17T20:44:52.2590646Z Generating XML reports... 2022-03-17T20:44:52.2659305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204448.xml 2022-03-17T20:44:54.8929022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:54.8940214Z 2022-03-17T20:44:54.8940760Z Running tests... 2022-03-17T20:44:54.8941950Z ---------------------------------------------------------------------- 2022-03-17T20:44:54.8960093Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:44:54.8961012Z 2022-03-17T20:44:54.8962656Z ---------------------------------------------------------------------- 2022-03-17T20:44:54.8963639Z Ran 1 test in 0.002s 2022-03-17T20:44:54.8964091Z 2022-03-17T20:44:54.8964368Z OK (skipped=1) 2022-03-17T20:44:54.8964726Z 2022-03-17T20:44:54.8965009Z Generating XML reports... 2022-03-17T20:44:54.9001428Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204454.xml 2022-03-17T20:44:57.3629832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:44:57.3644393Z 2022-03-17T20:44:57.3644597Z Running tests... 2022-03-17T20:44:57.3645396Z ---------------------------------------------------------------------- 2022-03-17T20:44:57.9639113Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126504 2022-03-17T20:44:57.9680956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126505 2022-03-17T20:44:57.9682533Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:44:59.0627767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:44:59.1102367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:44:59.1103387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:59.1135582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:44:59.1144792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:44:59.1148356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:44:59.2118030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:44:59.2119529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:44:59.2121779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:44:59.2173141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:44:59.6036622Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:44:59.6670674Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:01.5770204Z ok (4.212s) 2022-03-17T20:45:01.5770604Z 2022-03-17T20:45:01.5771571Z ---------------------------------------------------------------------- 2022-03-17T20:45:01.5772345Z Ran 1 test in 4.212s 2022-03-17T20:45:01.5772733Z 2022-03-17T20:45:01.5772919Z OK 2022-03-17T20:45:01.5773214Z 2022-03-17T20:45:01.5773488Z Generating XML reports... 2022-03-17T20:45:01.5853855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204457.xml 2022-03-17T20:45:04.1778775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:04.1793987Z 2022-03-17T20:45:04.1794289Z Running tests... 2022-03-17T20:45:04.1795534Z ---------------------------------------------------------------------- 2022-03-17T20:45:04.1812680Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:45:04.1813421Z 2022-03-17T20:45:04.1814236Z ---------------------------------------------------------------------- 2022-03-17T20:45:04.1815040Z Ran 1 test in 0.002s 2022-03-17T20:45:04.1815416Z 2022-03-17T20:45:04.1815653Z OK (skipped=1) 2022-03-17T20:45:04.1815993Z 2022-03-17T20:45:04.1816269Z Generating XML reports... 2022-03-17T20:45:04.1852016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204504.xml 2022-03-17T20:45:07.0469103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:07.0487123Z 2022-03-17T20:45:07.0487394Z Running tests... 2022-03-17T20:45:07.0487917Z ---------------------------------------------------------------------- 2022-03-17T20:45:07.6488153Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126796 2022-03-17T20:45:07.6528099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126797 2022-03-17T20:45:07.6529447Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:08.7124282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:08.7774023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:08.7774766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:08.7833792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:08.7834702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:08.8785098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:11.2639225Z skip (4.215s) 2022-03-17T20:45:11.2639644Z 2022-03-17T20:45:11.2640523Z ---------------------------------------------------------------------- 2022-03-17T20:45:11.2645188Z Ran 1 test in 4.215s 2022-03-17T20:45:11.2645602Z 2022-03-17T20:45:11.2645853Z OK (skipped=1) 2022-03-17T20:45:11.2646213Z 2022-03-17T20:45:11.2646501Z Generating XML reports... 2022-03-17T20:45:11.2728662Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204507.xml 2022-03-17T20:45:14.0504302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:14.0518462Z 2022-03-17T20:45:14.0518707Z Running tests... 2022-03-17T20:45:14.0519147Z ---------------------------------------------------------------------- 2022-03-17T20:45:14.0536519Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:45:14.0538266Z 2022-03-17T20:45:14.0538907Z ---------------------------------------------------------------------- 2022-03-17T20:45:14.0539293Z Ran 1 test in 0.002s 2022-03-17T20:45:14.0539470Z 2022-03-17T20:45:14.0539584Z OK (skipped=1) 2022-03-17T20:45:14.0539747Z 2022-03-17T20:45:14.0539876Z Generating XML reports... 2022-03-17T20:45:14.0579565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204514.xml 2022-03-17T20:45:16.5807607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:16.5820494Z 2022-03-17T20:45:16.5820736Z Running tests... 2022-03-17T20:45:16.5821372Z ---------------------------------------------------------------------- 2022-03-17T20:45:16.5840120Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:45:16.5840403Z 2022-03-17T20:45:16.5840865Z ---------------------------------------------------------------------- 2022-03-17T20:45:16.5841216Z Ran 1 test in 0.002s 2022-03-17T20:45:16.5841393Z 2022-03-17T20:45:16.5841505Z OK (skipped=1) 2022-03-17T20:45:16.5841668Z 2022-03-17T20:45:16.5841791Z Generating XML reports... 2022-03-17T20:45:16.5880300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204516.xml 2022-03-17T20:45:19.0459067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:19.0473797Z 2022-03-17T20:45:19.0474130Z Running tests... 2022-03-17T20:45:19.0474573Z ---------------------------------------------------------------------- 2022-03-17T20:45:19.6449190Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127138 2022-03-17T20:45:19.6489643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127139 2022-03-17T20:45:19.6491402Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:20.7563092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:20.7692253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:20.7694222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:20.7767036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:20.7778091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:20.8706784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:21.1652669Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:21.2547469Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:23.2585673Z ok (4.211s) 2022-03-17T20:45:23.2586109Z 2022-03-17T20:45:23.2587080Z ---------------------------------------------------------------------- 2022-03-17T20:45:23.2587841Z Ran 1 test in 4.211s 2022-03-17T20:45:23.2588212Z 2022-03-17T20:45:23.2588415Z OK 2022-03-17T20:45:23.2588704Z 2022-03-17T20:45:23.2588990Z Generating XML reports... 2022-03-17T20:45:23.2663252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204519.xml 2022-03-17T20:45:25.8140791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:25.8154462Z 2022-03-17T20:45:25.8154748Z Running tests... 2022-03-17T20:45:25.8155186Z ---------------------------------------------------------------------- 2022-03-17T20:45:26.4246262Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127353 2022-03-17T20:45:26.4320102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127354 2022-03-17T20:45:26.4321256Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:27.5335701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:27.5552274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:27.5554299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:27.5639833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:27.5641270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:27.6560262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:28.0369378Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:28.0827849Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:29.9410860Z ok (4.125s) 2022-03-17T20:45:29.9411377Z 2022-03-17T20:45:29.9412557Z ---------------------------------------------------------------------- 2022-03-17T20:45:29.9413511Z Ran 1 test in 4.125s 2022-03-17T20:45:29.9413994Z 2022-03-17T20:45:29.9414235Z OK 2022-03-17T20:45:29.9414596Z 2022-03-17T20:45:29.9414937Z Generating XML reports... 2022-03-17T20:45:29.9498218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204525.xml 2022-03-17T20:45:32.5328121Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:32.5340669Z 2022-03-17T20:45:32.5340948Z Running tests... 2022-03-17T20:45:32.5341682Z ---------------------------------------------------------------------- 2022-03-17T20:45:32.5355175Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:45:32.5355663Z 2022-03-17T20:45:32.5356376Z ---------------------------------------------------------------------- 2022-03-17T20:45:32.5357186Z Ran 1 test in 0.002s 2022-03-17T20:45:32.5357576Z 2022-03-17T20:45:32.5357812Z OK (skipped=1) 2022-03-17T20:45:32.5358153Z 2022-03-17T20:45:32.5358447Z Generating XML reports... 2022-03-17T20:45:32.5397047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204532.xml 2022-03-17T20:45:35.0088982Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:35.0107450Z 2022-03-17T20:45:35.0107752Z Running tests... 2022-03-17T20:45:35.0108191Z ---------------------------------------------------------------------- 2022-03-17T20:45:35.6474320Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127635 2022-03-17T20:45:35.6520271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127636 2022-03-17T20:45:35.6521504Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:36.7399195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:36.7431310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:36.7432315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:36.7500353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:36.7501303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:36.7503883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:45:36.8441790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:36.8442497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:45:36.8443355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:36.8524220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:37.2298181Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:37.2851000Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:39.0610547Z ok (4.050s) 2022-03-17T20:45:39.0610898Z 2022-03-17T20:45:39.0611479Z ---------------------------------------------------------------------- 2022-03-17T20:45:39.0611827Z Ran 1 test in 4.051s 2022-03-17T20:45:39.0612002Z 2022-03-17T20:45:39.0613523Z OK 2022-03-17T20:45:39.0613951Z 2022-03-17T20:45:39.0614231Z Generating XML reports... 2022-03-17T20:45:39.0699958Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204535.xml 2022-03-17T20:45:42.0483992Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:42.0499573Z 2022-03-17T20:45:42.0499933Z Running tests... 2022-03-17T20:45:42.0500928Z ---------------------------------------------------------------------- 2022-03-17T20:45:42.0516658Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:45:42.0517273Z 2022-03-17T20:45:42.0518441Z ---------------------------------------------------------------------- 2022-03-17T20:45:42.0519257Z Ran 1 test in 0.002s 2022-03-17T20:45:42.0519626Z 2022-03-17T20:45:42.0519851Z OK (skipped=1) 2022-03-17T20:45:42.0520202Z 2022-03-17T20:45:42.0520479Z Generating XML reports... 2022-03-17T20:45:42.0557557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204542.xml 2022-03-17T20:45:44.5233963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:44.5247947Z 2022-03-17T20:45:44.5248326Z Running tests... 2022-03-17T20:45:44.5249280Z ---------------------------------------------------------------------- 2022-03-17T20:45:45.1589718Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127921 2022-03-17T20:45:45.1636885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127922 2022-03-17T20:45:45.1638026Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:46.2782090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:46.3178656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:46.3180659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:46.3189137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:46.3204821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:46.4190617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:48.2726319Z skip (3.747s) 2022-03-17T20:45:48.2726814Z 2022-03-17T20:45:48.2727865Z ---------------------------------------------------------------------- 2022-03-17T20:45:48.2728732Z Ran 1 test in 3.748s 2022-03-17T20:45:48.2729145Z 2022-03-17T20:45:48.2729402Z OK (skipped=1) 2022-03-17T20:45:48.2729774Z 2022-03-17T20:45:48.2730073Z Generating XML reports... 2022-03-17T20:45:48.2813859Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204544.xml 2022-03-17T20:45:50.9099230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:50.9114869Z 2022-03-17T20:45:50.9115407Z Running tests... 2022-03-17T20:45:50.9116595Z ---------------------------------------------------------------------- 2022-03-17T20:45:51.5128373Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128129 2022-03-17T20:45:51.5167667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128130 2022-03-17T20:45:51.5169043Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:52.5792054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:52.6349104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:52.6351098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:52.6401375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:52.6405910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:52.7364916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:53.1752846Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:53.4139614Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:45:53.5988290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:45:53.6153597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:45:53.6155566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:53.6193049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:55.6286565Z ok (4.717s) 2022-03-17T20:45:55.6286977Z 2022-03-17T20:45:55.6287917Z ---------------------------------------------------------------------- 2022-03-17T20:45:55.6288686Z Ran 1 test in 4.717s 2022-03-17T20:45:55.6289047Z 2022-03-17T20:45:55.6289235Z OK 2022-03-17T20:45:55.6289527Z 2022-03-17T20:45:55.6289826Z Generating XML reports... 2022-03-17T20:45:55.6371130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204550.xml 2022-03-17T20:45:58.2043635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:58.2056691Z 2022-03-17T20:45:58.2057242Z Running tests... 2022-03-17T20:45:58.2058419Z ---------------------------------------------------------------------- 2022-03-17T20:45:58.8354225Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128355 2022-03-17T20:45:58.8405405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128356 2022-03-17T20:45:58.8406765Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:45:59.9802752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:00.0087662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:00.0089455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:00.0111174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:00.0117371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:00.1105771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:01.7512354Z skip (3.545s) 2022-03-17T20:46:01.7512634Z 2022-03-17T20:46:01.7513198Z ---------------------------------------------------------------------- 2022-03-17T20:46:01.7513739Z Ran 1 test in 3.545s 2022-03-17T20:46:01.7514004Z 2022-03-17T20:46:01.7514176Z OK (skipped=1) 2022-03-17T20:46:01.7514407Z 2022-03-17T20:46:01.7514596Z Generating XML reports... 2022-03-17T20:46:01.7590796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204558.xml 2022-03-17T20:46:04.4599924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:04.4614155Z 2022-03-17T20:46:04.4614655Z Running tests... 2022-03-17T20:46:04.4615787Z ---------------------------------------------------------------------- 2022-03-17T20:46:04.4630634Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:04.4631360Z 2022-03-17T20:46:04.4632456Z ---------------------------------------------------------------------- 2022-03-17T20:46:04.4633219Z Ran 1 test in 0.002s 2022-03-17T20:46:04.4633582Z 2022-03-17T20:46:04.4633810Z OK (skipped=1) 2022-03-17T20:46:04.4634161Z 2022-03-17T20:46:04.4634428Z Generating XML reports... 2022-03-17T20:46:04.4674080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204604.xml 2022-03-17T20:46:07.4565613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:07.4578436Z 2022-03-17T20:46:07.4579139Z Running tests... 2022-03-17T20:46:07.4580277Z ---------------------------------------------------------------------- 2022-03-17T20:46:07.4594970Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:07.4595595Z 2022-03-17T20:46:07.4596685Z ---------------------------------------------------------------------- 2022-03-17T20:46:07.4597447Z Ran 1 test in 0.002s 2022-03-17T20:46:07.4597818Z 2022-03-17T20:46:07.4598065Z OK (skipped=1) 2022-03-17T20:46:07.4598422Z 2022-03-17T20:46:07.4598695Z Generating XML reports... 2022-03-17T20:46:07.4644019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204607.xml 2022-03-17T20:46:10.1350623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:10.1366489Z 2022-03-17T20:46:10.1366893Z Running tests... 2022-03-17T20:46:10.1367925Z ---------------------------------------------------------------------- 2022-03-17T20:46:10.7351784Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128697 2022-03-17T20:46:10.7379300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128698 2022-03-17T20:46:10.7381121Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:46:11.7937918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:11.8741432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:11.8743035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:11.8753347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:11.8769328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:11.9749993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:12.2627019Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:46:12.3507418Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:46:14.9493417Z ok (4.812s) 2022-03-17T20:46:14.9493844Z 2022-03-17T20:46:14.9495385Z ---------------------------------------------------------------------- 2022-03-17T20:46:14.9496174Z Ran 1 test in 4.813s 2022-03-17T20:46:14.9496548Z 2022-03-17T20:46:14.9496757Z OK 2022-03-17T20:46:14.9497048Z 2022-03-17T20:46:14.9497345Z Generating XML reports... 2022-03-17T20:46:14.9580015Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204610.xml 2022-03-17T20:46:17.5765924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:17.5783167Z 2022-03-17T20:46:17.5783469Z Running tests... 2022-03-17T20:46:17.5783894Z ---------------------------------------------------------------------- 2022-03-17T20:46:17.5800153Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:17.5800402Z 2022-03-17T20:46:17.5800698Z ---------------------------------------------------------------------- 2022-03-17T20:46:17.5801104Z Ran 1 test in 0.002s 2022-03-17T20:46:17.5801353Z 2022-03-17T20:46:17.5801450Z OK (skipped=1) 2022-03-17T20:46:17.5801583Z 2022-03-17T20:46:17.5801676Z Generating XML reports... 2022-03-17T20:46:17.5840523Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204617.xml 2022-03-17T20:46:20.1130502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:20.1144471Z 2022-03-17T20:46:20.1145012Z Running tests... 2022-03-17T20:46:20.1146200Z ---------------------------------------------------------------------- 2022-03-17T20:46:20.7127393Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128979 2022-03-17T20:46:20.7162190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128980 2022-03-17T20:46:20.7163589Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:46:21.8409270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:21.8479335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:21.8480198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:21.8515126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:21.8519101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:21.9493158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:23.7261985Z skip (3.611s) 2022-03-17T20:46:23.7262398Z 2022-03-17T20:46:23.7263513Z ---------------------------------------------------------------------- 2022-03-17T20:46:23.7264394Z Ran 1 test in 3.612s 2022-03-17T20:46:23.7264756Z 2022-03-17T20:46:23.7264982Z OK (skipped=1) 2022-03-17T20:46:23.7265328Z 2022-03-17T20:46:23.7265615Z Generating XML reports... 2022-03-17T20:46:23.7342698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204620.xml 2022-03-17T20:46:26.3718891Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:26.3732302Z 2022-03-17T20:46:26.3732761Z Running tests... 2022-03-17T20:46:26.3733936Z ---------------------------------------------------------------------- 2022-03-17T20:46:26.3749815Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:26.3750511Z 2022-03-17T20:46:26.3751315Z ---------------------------------------------------------------------- 2022-03-17T20:46:26.3752107Z Ran 1 test in 0.002s 2022-03-17T20:46:26.3752459Z 2022-03-17T20:46:26.3752709Z OK (skipped=1) 2022-03-17T20:46:26.3753060Z 2022-03-17T20:46:26.3753340Z Generating XML reports... 2022-03-17T20:46:26.3791782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204626.xml 2022-03-17T20:46:28.9542260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:28.9555325Z 2022-03-17T20:46:28.9555825Z Running tests... 2022-03-17T20:46:28.9557014Z ---------------------------------------------------------------------- 2022-03-17T20:46:29.5692708Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129254 2022-03-17T20:46:29.5738649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129255 2022-03-17T20:46:29.5740051Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:46:30.6933439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:30.7502306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:30.7504514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:30.7544496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:30.7558477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:30.8510777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:32.5858560Z skip (3.630s) 2022-03-17T20:46:32.5858980Z 2022-03-17T20:46:32.5859836Z ---------------------------------------------------------------------- 2022-03-17T20:46:32.5860640Z Ran 1 test in 3.630s 2022-03-17T20:46:32.5860995Z 2022-03-17T20:46:32.5861243Z OK (skipped=1) 2022-03-17T20:46:32.5861611Z 2022-03-17T20:46:32.5861892Z Generating XML reports... 2022-03-17T20:46:32.5945485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204628.xml 2022-03-17T20:46:35.1751393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:35.1765438Z 2022-03-17T20:46:35.1765782Z Running tests... 2022-03-17T20:46:35.1766324Z ---------------------------------------------------------------------- 2022-03-17T20:46:35.1785479Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:35.1785769Z 2022-03-17T20:46:35.1786263Z ---------------------------------------------------------------------- 2022-03-17T20:46:35.1786899Z Ran 1 test in 0.002s 2022-03-17T20:46:35.1787089Z 2022-03-17T20:46:35.1787217Z OK (skipped=1) 2022-03-17T20:46:35.1787414Z 2022-03-17T20:46:35.1787542Z Generating XML reports... 2022-03-17T20:46:35.1825839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204635.xml 2022-03-17T20:46:37.6817282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:37.6830443Z 2022-03-17T20:46:37.6830807Z Running tests... 2022-03-17T20:46:37.6831296Z ---------------------------------------------------------------------- 2022-03-17T20:46:37.6853549Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:37.6854060Z 2022-03-17T20:46:37.6855055Z ---------------------------------------------------------------------- 2022-03-17T20:46:37.6855899Z Ran 1 test in 0.002s 2022-03-17T20:46:37.6856312Z 2022-03-17T20:46:37.6856539Z OK (skipped=1) 2022-03-17T20:46:37.6856923Z 2022-03-17T20:46:37.6857202Z Generating XML reports... 2022-03-17T20:46:37.6896361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204637.xml 2022-03-17T20:46:40.2861091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:40.2873077Z 2022-03-17T20:46:40.2873547Z Running tests... 2022-03-17T20:46:40.2874387Z ---------------------------------------------------------------------- 2022-03-17T20:46:40.2892031Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:40.2892350Z 2022-03-17T20:46:40.2892714Z ---------------------------------------------------------------------- 2022-03-17T20:46:40.2893063Z Ran 1 test in 0.002s 2022-03-17T20:46:40.2893237Z 2022-03-17T20:46:40.2893344Z OK (skipped=1) 2022-03-17T20:46:40.2893514Z 2022-03-17T20:46:40.2893646Z Generating XML reports... 2022-03-17T20:46:40.2936749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204640.xml 2022-03-17T20:46:43.0778677Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:43.0792086Z 2022-03-17T20:46:43.0792314Z Running tests... 2022-03-17T20:46:43.0793085Z ---------------------------------------------------------------------- 2022-03-17T20:46:43.0815703Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:43.0815956Z 2022-03-17T20:46:43.0816436Z ---------------------------------------------------------------------- 2022-03-17T20:46:43.0816733Z Ran 1 test in 0.003s 2022-03-17T20:46:43.0816876Z 2022-03-17T20:46:43.0816964Z OK (skipped=1) 2022-03-17T20:46:43.0817088Z 2022-03-17T20:46:43.0817201Z Generating XML reports... 2022-03-17T20:46:43.0857565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204643.xml 2022-03-17T20:46:45.6916543Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:45.6931955Z 2022-03-17T20:46:45.6932294Z Running tests... 2022-03-17T20:46:45.6932790Z ---------------------------------------------------------------------- 2022-03-17T20:46:45.6957389Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:45.6957655Z 2022-03-17T20:46:45.6957985Z ---------------------------------------------------------------------- 2022-03-17T20:46:45.6958325Z Ran 1 test in 0.003s 2022-03-17T20:46:45.6958513Z 2022-03-17T20:46:45.6958623Z OK (skipped=1) 2022-03-17T20:46:45.6958849Z 2022-03-17T20:46:45.6959006Z Generating XML reports... 2022-03-17T20:46:45.7002289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204645.xml 2022-03-17T20:46:48.4253673Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:48.4266341Z 2022-03-17T20:46:48.4266775Z Running tests... 2022-03-17T20:46:48.4267692Z ---------------------------------------------------------------------- 2022-03-17T20:46:49.0314359Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129797 2022-03-17T20:46:49.0356378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129798 2022-03-17T20:46:49.0357731Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:46:50.1117897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:50.1661972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:50.1663900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:50.1728245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:50.1729573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:50.2674804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:50.2848317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:46:50.2849573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:46:50.2851362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:50.2853104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:50.2854414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:46:50.2954193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:46:50.2956027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:46:50.2958126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:46:52.8464070Z ok (4.419s) 2022-03-17T20:46:52.8464507Z 2022-03-17T20:46:52.8466504Z ---------------------------------------------------------------------- 2022-03-17T20:46:52.8467413Z Ran 1 test in 4.419s 2022-03-17T20:46:52.8467816Z 2022-03-17T20:46:52.8468022Z OK 2022-03-17T20:46:52.8468322Z 2022-03-17T20:46:52.8468642Z Generating XML reports... 2022-03-17T20:46:52.8544830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204648.xml 2022-03-17T20:46:55.4063735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:55.4076436Z 2022-03-17T20:46:55.4076963Z Running tests... 2022-03-17T20:46:55.4078139Z ---------------------------------------------------------------------- 2022-03-17T20:46:56.0087974Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130015 2022-03-17T20:46:56.0126210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130016 2022-03-17T20:46:56.0127518Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:46:57.1217895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:57.1411662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:57.1414496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:57.1421100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:57.1431884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:57.2428604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:57.6282334Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:46:57.7434017Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:46:59.6229023Z ok (4.215s) 2022-03-17T20:46:59.6229412Z 2022-03-17T20:46:59.6230376Z ---------------------------------------------------------------------- 2022-03-17T20:46:59.6231148Z Ran 1 test in 4.215s 2022-03-17T20:46:59.6231532Z 2022-03-17T20:46:59.6231715Z OK 2022-03-17T20:46:59.6232007Z 2022-03-17T20:46:59.6236533Z Generating XML reports... 2022-03-17T20:46:59.6305033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204655.xml 2022-03-17T20:47:02.2595076Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:02.2608536Z 2022-03-17T20:47:02.2608898Z Running tests... 2022-03-17T20:47:02.2609433Z ---------------------------------------------------------------------- 2022-03-17T20:47:02.8603217Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130238 2022-03-17T20:47:02.8636697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130239 2022-03-17T20:47:02.8637291Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:03.9923193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:04.0062051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:04.0064008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:04.0126113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:04.0135372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:04.1077876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:05.8735021Z skip (3.612s) 2022-03-17T20:47:05.8735823Z 2022-03-17T20:47:05.8737176Z ---------------------------------------------------------------------- 2022-03-17T20:47:05.8738028Z Ran 1 test in 3.612s 2022-03-17T20:47:05.8738376Z 2022-03-17T20:47:05.8738612Z OK (skipped=1) 2022-03-17T20:47:05.8738988Z 2022-03-17T20:47:05.8739259Z Generating XML reports... 2022-03-17T20:47:05.8818264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204702.xml 2022-03-17T20:47:08.4174328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:08.4191798Z 2022-03-17T20:47:08.4192279Z Running tests... 2022-03-17T20:47:08.4193073Z ---------------------------------------------------------------------- 2022-03-17T20:47:09.0183803Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130446 2022-03-17T20:47:09.0214257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130447 2022-03-17T20:47:09.0215892Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:10.1448256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:10.1639232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:10.1640833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:10.1653847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:10.1660257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:10.2655043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:12.4312157Z ok (4.012s) 2022-03-17T20:47:12.4312527Z 2022-03-17T20:47:12.4313927Z ---------------------------------------------------------------------- 2022-03-17T20:47:12.4314800Z Ran 1 test in 4.012s 2022-03-17T20:47:12.4315200Z 2022-03-17T20:47:12.4317427Z OK 2022-03-17T20:47:12.4317754Z 2022-03-17T20:47:12.4318056Z Generating XML reports... 2022-03-17T20:47:12.4393138Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204708.xml 2022-03-17T20:47:14.9750790Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:14.9765794Z 2022-03-17T20:47:14.9766251Z Running tests... 2022-03-17T20:47:14.9767408Z ---------------------------------------------------------------------- 2022-03-17T20:47:15.5728520Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130654 2022-03-17T20:47:15.5772358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130655 2022-03-17T20:47:15.5774552Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:16.7023622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:16.7138928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:16.7141614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:16.7228153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:16.7240510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:16.8153610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:18.5874441Z ok (3.610s) 2022-03-17T20:47:18.5874939Z 2022-03-17T20:47:18.5876263Z ---------------------------------------------------------------------- 2022-03-17T20:47:18.5877155Z Ran 1 test in 3.611s 2022-03-17T20:47:18.5877570Z 2022-03-17T20:47:18.5877769Z OK 2022-03-17T20:47:18.5878077Z 2022-03-17T20:47:18.5878678Z Generating XML reports... 2022-03-17T20:47:18.5949783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204714.xml 2022-03-17T20:47:21.2043252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:21.2055849Z 2022-03-17T20:47:21.2056392Z Running tests... 2022-03-17T20:47:21.2057688Z ---------------------------------------------------------------------- 2022-03-17T20:47:21.8067166Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130862 2022-03-17T20:47:21.8105696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130863 2022-03-17T20:47:21.8107430Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:22.9300361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:22.9331641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:22.9332760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:22.9401817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:22.9404690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:23.0346644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:23.4154546Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:47:25.2203466Z ok (4.014s) 2022-03-17T20:47:25.2204000Z 2022-03-17T20:47:25.2205031Z ---------------------------------------------------------------------- 2022-03-17T20:47:25.2205846Z Ran 1 test in 4.014s 2022-03-17T20:47:25.2206215Z 2022-03-17T20:47:25.2206410Z OK 2022-03-17T20:47:25.2206746Z 2022-03-17T20:47:25.2207041Z Generating XML reports... 2022-03-17T20:47:25.2291055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204721.xml 2022-03-17T20:47:27.7677499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:27.7693922Z 2022-03-17T20:47:27.7694369Z Running tests... 2022-03-17T20:47:27.7695526Z ---------------------------------------------------------------------- 2022-03-17T20:47:28.4009222Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 302 2022-03-17T20:47:28.4048746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 303 2022-03-17T20:47:28.4050075Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:29.5305925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:29.5460047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:29.5462261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:29.5511144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:29.5516910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:29.6470390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:31.3144887Z ok (3.545s) 2022-03-17T20:47:31.3145473Z 2022-03-17T20:47:31.3146493Z ---------------------------------------------------------------------- 2022-03-17T20:47:31.3147480Z Ran 1 test in 3.545s 2022-03-17T20:47:31.3147939Z 2022-03-17T20:47:31.3148200Z OK 2022-03-17T20:47:31.3148539Z 2022-03-17T20:47:31.3148839Z Generating XML reports... 2022-03-17T20:47:31.3226010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204727.xml 2022-03-17T20:47:33.8778549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:33.8791223Z 2022-03-17T20:47:33.8791354Z Running tests... 2022-03-17T20:47:33.8792087Z ---------------------------------------------------------------------- 2022-03-17T20:47:33.8804528Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:47:33.8804894Z 2022-03-17T20:47:33.8805638Z ---------------------------------------------------------------------- 2022-03-17T20:47:33.8806083Z Ran 1 test in 0.002s 2022-03-17T20:47:33.8806283Z 2022-03-17T20:47:33.8806383Z OK (skipped=1) 2022-03-17T20:47:33.8806547Z 2022-03-17T20:47:33.8808498Z Generating XML reports... 2022-03-17T20:47:33.8845328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204733.xml 2022-03-17T20:47:36.3637469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:36.3652887Z 2022-03-17T20:47:36.3653820Z Running tests... 2022-03-17T20:47:36.3654552Z ---------------------------------------------------------------------- 2022-03-17T20:47:37.0068008Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 577 2022-03-17T20:47:37.0106969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 578 2022-03-17T20:47:37.0108325Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:38.0890562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:38.1003006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:38.1004164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:38.1090462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:38.1092697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:38.2013308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:38.5829574Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:47:38.6370883Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:47:40.5202932Z ok (4.154s) 2022-03-17T20:47:40.5203429Z 2022-03-17T20:47:40.5204534Z ---------------------------------------------------------------------- 2022-03-17T20:47:40.5205326Z Ran 1 test in 4.155s 2022-03-17T20:47:40.5205680Z 2022-03-17T20:47:40.5205881Z OK 2022-03-17T20:47:40.5206190Z 2022-03-17T20:47:40.5206480Z Generating XML reports... 2022-03-17T20:47:40.5286674Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204736.xml 2022-03-17T20:47:43.2338396Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:43.2353143Z 2022-03-17T20:47:43.2353444Z Running tests... 2022-03-17T20:47:43.2354025Z ---------------------------------------------------------------------- 2022-03-17T20:47:43.2368841Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:47:43.2369202Z 2022-03-17T20:47:43.2369561Z ---------------------------------------------------------------------- 2022-03-17T20:47:43.2369927Z Ran 1 test in 0.002s 2022-03-17T20:47:43.2370100Z 2022-03-17T20:47:43.2370196Z OK (skipped=1) 2022-03-17T20:47:43.2370361Z 2022-03-17T20:47:43.2370486Z Generating XML reports... 2022-03-17T20:47:43.2412430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204743.xml 2022-03-17T20:47:45.9811763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:45.9822310Z 2022-03-17T20:47:45.9822908Z Running tests... 2022-03-17T20:47:45.9824114Z ---------------------------------------------------------------------- 2022-03-17T20:47:45.9839493Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:47:45.9840365Z 2022-03-17T20:47:45.9840976Z ---------------------------------------------------------------------- 2022-03-17T20:47:45.9841617Z Ran 1 test in 0.002s 2022-03-17T20:47:45.9841758Z 2022-03-17T20:47:45.9841947Z OK (skipped=1) 2022-03-17T20:47:45.9842100Z 2022-03-17T20:47:45.9842209Z Generating XML reports... 2022-03-17T20:47:45.9879906Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204745.xml 2022-03-17T20:47:48.5912020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:48.5922980Z 2022-03-17T20:47:48.5923532Z Running tests... 2022-03-17T20:47:48.5924717Z ---------------------------------------------------------------------- 2022-03-17T20:47:48.5941226Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:47:48.5942146Z 2022-03-17T20:47:48.5943182Z ---------------------------------------------------------------------- 2022-03-17T20:47:48.5944032Z Ran 1 test in 0.002s 2022-03-17T20:47:48.5944424Z 2022-03-17T20:47:48.5944657Z OK (skipped=1) 2022-03-17T20:47:48.5945022Z 2022-03-17T20:47:48.5945276Z Generating XML reports... 2022-03-17T20:47:48.5981357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204748.xml 2022-03-17T20:47:51.1438207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:51.1450149Z 2022-03-17T20:47:51.1450464Z Running tests... 2022-03-17T20:47:51.1450998Z ---------------------------------------------------------------------- 2022-03-17T20:47:51.7642706Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1001 2022-03-17T20:47:51.7676167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1002 2022-03-17T20:47:51.7677790Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:52.8232431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:52.9233533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:52.9234261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:52.9239447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:52.9243905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:53.0248780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:53.6278850Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:47:53.6929602Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:47:55.4773837Z ok (4.332s) 2022-03-17T20:47:55.4774132Z 2022-03-17T20:47:55.4775041Z ---------------------------------------------------------------------- 2022-03-17T20:47:55.4775635Z Ran 1 test in 4.332s 2022-03-17T20:47:55.4775807Z 2022-03-17T20:47:55.4775899Z OK 2022-03-17T20:47:55.4776566Z 2022-03-17T20:47:55.4776722Z Generating XML reports... 2022-03-17T20:47:55.4854956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204751.xml 2022-03-17T20:47:58.1216979Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:58.1230679Z 2022-03-17T20:47:58.1231025Z Running tests... 2022-03-17T20:47:58.1231559Z ---------------------------------------------------------------------- 2022-03-17T20:47:58.7492428Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1216 2022-03-17T20:47:58.7544737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1217 2022-03-17T20:47:58.7546057Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:47:59.8300590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:59.9048097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:59.9049995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:59.9108848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:59.9112370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:00.0055927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:01.6640889Z skip (3.540s) 2022-03-17T20:48:01.6641320Z 2022-03-17T20:48:01.6643004Z ---------------------------------------------------------------------- 2022-03-17T20:48:01.6644122Z Ran 1 test in 3.541s 2022-03-17T20:48:01.6644479Z 2022-03-17T20:48:01.6644719Z OK (skipped=1) 2022-03-17T20:48:01.6645086Z 2022-03-17T20:48:01.6645362Z Generating XML reports... 2022-03-17T20:48:01.6728199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204758.xml 2022-03-17T20:48:04.2632978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:04.2645203Z 2022-03-17T20:48:04.2645731Z Running tests... 2022-03-17T20:48:04.2646865Z ---------------------------------------------------------------------- 2022-03-17T20:48:04.8637092Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1424 2022-03-17T20:48:04.8666891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1425 2022-03-17T20:48:04.8668230Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:48:05.9393329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:05.9877692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:05.9878945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:05.9902287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:05.9902884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:06.0906408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:07.8744614Z skip (3.609s) 2022-03-17T20:48:07.8745102Z 2022-03-17T20:48:07.8746151Z ---------------------------------------------------------------------- 2022-03-17T20:48:07.8746943Z Ran 1 test in 3.610s 2022-03-17T20:48:07.8747317Z 2022-03-17T20:48:07.8747554Z OK (skipped=1) 2022-03-17T20:48:07.8747890Z 2022-03-17T20:48:07.8748168Z Generating XML reports... 2022-03-17T20:48:07.8821771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204804.xml 2022-03-17T20:48:10.5375375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:10.5387377Z 2022-03-17T20:48:10.5387655Z Running tests... 2022-03-17T20:48:10.5388186Z ---------------------------------------------------------------------- 2022-03-17T20:48:11.1633858Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1632 2022-03-17T20:48:11.1671360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1633 2022-03-17T20:48:11.1672692Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:48:12.2449642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:12.3348938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:12.3350115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:12.3362813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:12.3372149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:12.4361343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:12.8204980Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:48:12.8745288Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:48:22.2203304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:22.2204912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:24.9950920Z ok (14.456s) 2022-03-17T20:48:24.9951544Z 2022-03-17T20:48:24.9952781Z ---------------------------------------------------------------------- 2022-03-17T20:48:24.9953783Z Ran 1 test in 14.456s 2022-03-17T20:48:24.9954175Z 2022-03-17T20:48:24.9954370Z OK 2022-03-17T20:48:24.9954649Z 2022-03-17T20:48:24.9954943Z Generating XML reports... 2022-03-17T20:48:25.0027558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204810.xml 2022-03-17T20:48:27.6584929Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:27.6592855Z 2022-03-17T20:48:27.6593057Z Running tests... 2022-03-17T20:48:27.6594309Z ---------------------------------------------------------------------- 2022-03-17T20:48:28.2593558Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1851 2022-03-17T20:48:28.2621971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1852 2022-03-17T20:48:28.2623322Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:48:29.3822591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:29.3986019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:29.3987950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:29.4023177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:29.4029859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:29.4997673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:29.8760925Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:48:29.9348075Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:48:39.2067286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:39.2068789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:39.2076980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:39.2078333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:41.8945227Z ok (14.234s) 2022-03-17T20:48:41.8945682Z 2022-03-17T20:48:41.8946865Z ---------------------------------------------------------------------- 2022-03-17T20:48:41.8947864Z Ran 1 test in 14.235s 2022-03-17T20:48:41.8948310Z 2022-03-17T20:48:41.8948505Z OK 2022-03-17T20:48:41.8948799Z 2022-03-17T20:48:41.8949082Z Generating XML reports... 2022-03-17T20:48:41.9026304Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204827.xml 2022-03-17T20:48:44.4844982Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:44.4854176Z 2022-03-17T20:48:44.4854575Z Running tests... 2022-03-17T20:48:44.4855972Z ---------------------------------------------------------------------- 2022-03-17T20:48:45.0856110Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2070 2022-03-17T20:48:45.0900406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2071 2022-03-17T20:48:45.0901961Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:48:46.2069118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:46.2247551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:46.2249702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:46.2270138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:46.2272110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:46.3260050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:46.6057673Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:48:46.7081632Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:48:55.9509693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9511096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9519704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9521174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9674445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9676045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9678410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:55.9679883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:48:58.7220278Z ok (14.236s) 2022-03-17T20:48:58.7220903Z 2022-03-17T20:48:58.7222228Z ---------------------------------------------------------------------- 2022-03-17T20:48:58.7223360Z Ran 1 test in 14.236s 2022-03-17T20:48:58.7223736Z 2022-03-17T20:48:58.7223929Z OK 2022-03-17T20:48:58.7224258Z 2022-03-17T20:48:58.7225253Z Generating XML reports... 2022-03-17T20:48:58.7303386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204844.xml 2022-03-17T20:49:01.3088140Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:01.3099155Z 2022-03-17T20:49:01.3099449Z Running tests... 2022-03-17T20:49:01.3099987Z ---------------------------------------------------------------------- 2022-03-17T20:49:01.9078572Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2289 2022-03-17T20:49:01.9115387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2290 2022-03-17T20:49:01.9116816Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:49:03.0442573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:03.0443537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:03.0444625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:03.0445460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:03.0447911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:03.0448804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:03.4332035Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:03.4987348Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:12.7439684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.7447429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.7465534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.7466493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.8324936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.8326287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.8330033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:12.8331975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:15.6440872Z ok (14.333s) 2022-03-17T20:49:15.6441425Z 2022-03-17T20:49:15.6442971Z ---------------------------------------------------------------------- 2022-03-17T20:49:15.6443982Z Ran 1 test in 14.334s 2022-03-17T20:49:15.6444432Z 2022-03-17T20:49:15.6444687Z OK 2022-03-17T20:49:15.6444983Z 2022-03-17T20:49:15.6445274Z Generating XML reports... 2022-03-17T20:49:15.6523439Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204901.xml 2022-03-17T20:49:18.1864357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:18.1877760Z 2022-03-17T20:49:18.1878059Z Running tests... 2022-03-17T20:49:18.1878489Z ---------------------------------------------------------------------- 2022-03-17T20:49:18.7894843Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2508 2022-03-17T20:49:18.7939636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2509 2022-03-17T20:49:18.7940953Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:49:19.9136353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:19.9148763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:19.9149934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:19.9240157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:19.9249709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:20.0164567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:20.3103807Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:20.3957685Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:20.4874758Z 2022-03-17T20:49:22.2031495Z ok (4.015s) 2022-03-17T20:49:22.2031926Z 2022-03-17T20:49:22.2033220Z ---------------------------------------------------------------------- 2022-03-17T20:49:22.2034056Z Ran 1 test in 4.015s 2022-03-17T20:49:22.2034433Z 2022-03-17T20:49:22.2034632Z OK 2022-03-17T20:49:22.2034912Z 2022-03-17T20:49:22.2035202Z Generating XML reports... 2022-03-17T20:49:22.2105203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204918.xml 2022-03-17T20:49:24.8295286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:24.8308752Z 2022-03-17T20:49:24.8309267Z Running tests... 2022-03-17T20:49:24.8310874Z ---------------------------------------------------------------------- 2022-03-17T20:49:25.4670323Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2723 2022-03-17T20:49:25.4701362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2724 2022-03-17T20:49:25.4703088Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:49:26.5422062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:26.6064756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:26.6065889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:26.6129847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:26.6130383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:26.7080119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:27.0920600Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:27.1429006Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:28.7802019Z ok (3.949s) 2022-03-17T20:49:28.7802514Z 2022-03-17T20:49:28.7803806Z ---------------------------------------------------------------------- 2022-03-17T20:49:28.7804585Z Ran 1 test in 3.949s 2022-03-17T20:49:28.7804970Z 2022-03-17T20:49:28.7805137Z OK 2022-03-17T20:49:28.7805425Z 2022-03-17T20:49:28.7805710Z Generating XML reports... 2022-03-17T20:49:28.7880038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204924.xml 2022-03-17T20:49:31.3626909Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:31.3640008Z 2022-03-17T20:49:31.3640444Z Running tests... 2022-03-17T20:49:31.3642038Z ---------------------------------------------------------------------- 2022-03-17T20:49:32.0097274Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2938 2022-03-17T20:49:32.0145361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2939 2022-03-17T20:49:32.0146902Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:49:33.0704588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:33.1126208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:33.1128532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:33.1214158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:33.1215730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:33.2140388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:35.0236718Z skip (3.659s) 2022-03-17T20:49:35.0237289Z 2022-03-17T20:49:35.0238292Z ---------------------------------------------------------------------- 2022-03-17T20:49:35.0239231Z Ran 1 test in 3.659s 2022-03-17T20:49:35.0239679Z 2022-03-17T20:49:35.0239973Z OK (skipped=1) 2022-03-17T20:49:35.0240357Z 2022-03-17T20:49:35.0240660Z Generating XML reports... 2022-03-17T20:49:35.0310315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204931.xml 2022-03-17T20:49:37.7198960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:37.7215068Z 2022-03-17T20:49:37.7215251Z Running tests... 2022-03-17T20:49:37.7215844Z ---------------------------------------------------------------------- 2022-03-17T20:49:38.3225857Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3146 2022-03-17T20:49:38.3264972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3147 2022-03-17T20:49:38.3265657Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:49:39.3995802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:39.4415646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:39.4417893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:39.4502802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:39.4504339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:39.5429477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:39.9235661Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:39.9739497Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:49.2899526Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:49:52.0522300Z ok (14.330s) 2022-03-17T20:49:52.0522708Z 2022-03-17T20:49:52.0524475Z ---------------------------------------------------------------------- 2022-03-17T20:49:52.0525237Z Ran 1 test in 14.331s 2022-03-17T20:49:52.0525601Z 2022-03-17T20:49:52.0525807Z OK 2022-03-17T20:49:52.0526101Z 2022-03-17T20:49:52.0526357Z Generating XML reports... 2022-03-17T20:49:52.0607421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204937.xml 2022-03-17T20:49:54.6362086Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:54.6374727Z 2022-03-17T20:49:54.6375044Z Running tests... 2022-03-17T20:49:54.6375695Z ---------------------------------------------------------------------- 2022-03-17T20:49:55.2748610Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3365 2022-03-17T20:49:55.2776858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3366 2022-03-17T20:49:55.2778427Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:49:56.3981520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:56.4254648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:56.4256999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:56.4285203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:56.4292272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:56.5267405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:56.9133219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:49:56.9689513Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:50:06.3220193Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:50:06.3224537Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:50:09.1113913Z ok (14.473s) 2022-03-17T20:50:09.1114467Z 2022-03-17T20:50:09.1115608Z ---------------------------------------------------------------------- 2022-03-17T20:50:09.1116605Z Ran 1 test in 14.474s 2022-03-17T20:50:09.1116981Z 2022-03-17T20:50:09.1117171Z OK 2022-03-17T20:50:09.1117479Z 2022-03-17T20:50:09.1117763Z Generating XML reports... 2022-03-17T20:50:09.1188181Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204954.xml 2022-03-17T20:50:11.6843609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:11.6858050Z 2022-03-17T20:50:11.6858301Z Running tests... 2022-03-17T20:50:11.6858835Z ---------------------------------------------------------------------- 2022-03-17T20:50:11.6881741Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:11.6882228Z 2022-03-17T20:50:11.6882781Z ---------------------------------------------------------------------- 2022-03-17T20:50:11.6883152Z Ran 1 test in 0.002s 2022-03-17T20:50:11.6883320Z 2022-03-17T20:50:11.6883416Z OK (skipped=1) 2022-03-17T20:50:11.6883577Z 2022-03-17T20:50:11.6883700Z Generating XML reports... 2022-03-17T20:50:11.6922727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205011.xml 2022-03-17T20:50:14.3613720Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:14.3628064Z 2022-03-17T20:50:14.3628309Z Running tests... 2022-03-17T20:50:14.3628831Z ---------------------------------------------------------------------- 2022-03-17T20:50:14.9679243Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3651 2022-03-17T20:50:14.9712486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3652 2022-03-17T20:50:14.9714348Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:50:16.0634775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:16.0841705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:16.0843985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:16.0938065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:16.0941676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:16.1851548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:16.5628915Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:50:16.6177513Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:50:25.8826977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:50:25.8828855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:50:28.6034305Z ok (14.240s) 2022-03-17T20:50:28.6034691Z 2022-03-17T20:50:28.6035870Z ---------------------------------------------------------------------- 2022-03-17T20:50:28.6036879Z Ran 1 test in 14.240s 2022-03-17T20:50:28.6037340Z 2022-03-17T20:50:28.6037537Z OK 2022-03-17T20:50:28.6037868Z 2022-03-17T20:50:28.6038187Z Generating XML reports... 2022-03-17T20:50:28.6108258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205014.xml 2022-03-17T20:50:31.1428971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:31.1444674Z 2022-03-17T20:50:31.1445033Z Running tests... 2022-03-17T20:50:31.1445466Z ---------------------------------------------------------------------- 2022-03-17T20:50:31.7463690Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3870 2022-03-17T20:50:31.7491898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3871 2022-03-17T20:50:31.7493600Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:50:32.8644623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:32.8923172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:32.8924147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:32.8945797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:32.8947942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:32.9937255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:33.2754300Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:50:33.3850469Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:50:33.5018782Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:50:33.5020027Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:50:33.5044344Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:50:33.5045448Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:50:45.3764423Z ok (14.231s) 2022-03-17T20:50:45.3765017Z 2022-03-17T20:50:45.3766049Z ---------------------------------------------------------------------- 2022-03-17T20:50:45.3766808Z Ran 1 test in 14.232s 2022-03-17T20:50:45.3767167Z 2022-03-17T20:50:45.3767393Z OK 2022-03-17T20:50:45.3768442Z 2022-03-17T20:50:45.3769600Z Generating XML reports... 2022-03-17T20:50:45.3847917Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205031.xml 2022-03-17T20:50:47.9654375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:47.9667935Z 2022-03-17T20:50:47.9668539Z Running tests... 2022-03-17T20:50:47.9670222Z ---------------------------------------------------------------------- 2022-03-17T20:50:48.5665430Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4151 2022-03-17T20:50:48.5698613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4152 2022-03-17T20:50:48.5699957Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:50:49.6373528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:49.6940705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:49.6941580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:49.6979818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:49.6980594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:49.7949678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:51.2766128Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:50:51.3442152Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:51:00.6155815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:00.6157778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:04.0010180Z ok (16.034s) 2022-03-17T20:51:04.0010723Z 2022-03-17T20:51:04.0012019Z ---------------------------------------------------------------------- 2022-03-17T20:51:04.0013019Z Ran 1 test in 16.034s 2022-03-17T20:51:04.0013458Z 2022-03-17T20:51:04.0013722Z OK 2022-03-17T20:51:04.0014167Z 2022-03-17T20:51:04.0014468Z Generating XML reports... 2022-03-17T20:51:04.0093345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205047.xml 2022-03-17T20:51:06.7058000Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:06.7073282Z 2022-03-17T20:51:06.7073669Z Running tests... 2022-03-17T20:51:06.7074096Z ---------------------------------------------------------------------- 2022-03-17T20:51:07.3275483Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4370 2022-03-17T20:51:07.3311269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4371 2022-03-17T20:51:07.3312576Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:51:08.4286132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:08.4816049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:08.4817879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:08.4892712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:08.4895022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:08.5830051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:09.0129206Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:51:09.0461678Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:51:18.1649312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:18.1650841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:20.7632212Z ok (14.055s) 2022-03-17T20:51:20.7632591Z 2022-03-17T20:51:20.7633552Z ---------------------------------------------------------------------- 2022-03-17T20:51:20.7634319Z Ran 1 test in 14.056s 2022-03-17T20:51:20.7634676Z 2022-03-17T20:51:20.7637839Z OK 2022-03-17T20:51:20.7640091Z 2022-03-17T20:51:20.7640542Z Generating XML reports... 2022-03-17T20:51:20.7709031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205106.xml 2022-03-17T20:51:23.3133326Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:23.3147316Z 2022-03-17T20:51:23.3147668Z Running tests... 2022-03-17T20:51:23.3148233Z ---------------------------------------------------------------------- 2022-03-17T20:51:23.9145555Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4589 2022-03-17T20:51:23.9177309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4590 2022-03-17T20:51:23.9178629Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:51:25.0545089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:25.0613692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:25.0615888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:25.0646199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:25.0649624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:25.1622446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:26.9272773Z skip (3.612s) 2022-03-17T20:51:26.9273157Z 2022-03-17T20:51:26.9275214Z ---------------------------------------------------------------------- 2022-03-17T20:51:26.9276346Z Ran 1 test in 3.612s 2022-03-17T20:51:26.9276820Z 2022-03-17T20:51:26.9277241Z OK (skipped=1) 2022-03-17T20:51:26.9277619Z 2022-03-17T20:51:26.9277908Z Generating XML reports... 2022-03-17T20:51:26.9347973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205123.xml 2022-03-17T20:51:29.5148174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:29.5161761Z 2022-03-17T20:51:29.5162225Z Running tests... 2022-03-17T20:51:30.1413188Z ---------------------------------------------------------------------- 2022-03-17T20:51:30.1414684Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4797 2022-03-17T20:51:30.1447603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4798 2022-03-17T20:51:30.1448957Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:51:31.2684614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:31.3065421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:31.3067375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:31.3088575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:31.3090865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:31.4081253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:33.1546954Z skip (3.638s) 2022-03-17T20:51:33.1547354Z 2022-03-17T20:51:33.1548176Z ---------------------------------------------------------------------- 2022-03-17T20:51:33.1548970Z Ran 1 test in 3.638s 2022-03-17T20:51:33.1549360Z 2022-03-17T20:51:33.1549595Z OK (skipped=1) 2022-03-17T20:51:33.1549964Z 2022-03-17T20:51:33.1550240Z Generating XML reports... 2022-03-17T20:51:33.1624788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205129.xml 2022-03-17T20:51:35.7150518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:35.7158537Z 2022-03-17T20:51:35.7158838Z Running tests... 2022-03-17T20:51:35.7160013Z ---------------------------------------------------------------------- 2022-03-17T20:51:36.3148271Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5005 2022-03-17T20:51:36.3178857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5006 2022-03-17T20:51:36.3180517Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:51:37.4421076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:37.4423145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:37.4425312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:37.4427541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:37.4428961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:37.4430151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:39.1270820Z skip (3.410s) 2022-03-17T20:51:39.1271474Z 2022-03-17T20:51:39.1272600Z ---------------------------------------------------------------------- 2022-03-17T20:51:39.1273591Z Ran 1 test in 3.411s 2022-03-17T20:51:39.1274030Z 2022-03-17T20:51:39.1274350Z OK (skipped=1) 2022-03-17T20:51:39.1274699Z 2022-03-17T20:51:39.1274965Z Generating XML reports... 2022-03-17T20:51:39.1348217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205135.xml 2022-03-17T20:51:41.6787152Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:41.6800250Z 2022-03-17T20:51:41.6800519Z Running tests... 2022-03-17T20:51:41.6801101Z ---------------------------------------------------------------------- 2022-03-17T20:51:42.2793155Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5213 2022-03-17T20:51:42.2826813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5214 2022-03-17T20:51:42.2828158Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:51:43.3500628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:43.4039427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:43.4041460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:43.4111811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:43.4118071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:43.5051123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:45.1922493Z skip (3.512s) 2022-03-17T20:51:45.1923053Z 2022-03-17T20:51:45.1924284Z ---------------------------------------------------------------------- 2022-03-17T20:51:45.1925281Z Ran 1 test in 3.512s 2022-03-17T20:51:45.1925715Z 2022-03-17T20:51:45.1925930Z OK (skipped=1) 2022-03-17T20:51:45.1926299Z 2022-03-17T20:51:45.1926573Z Generating XML reports... 2022-03-17T20:51:45.2006939Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205141.xml 2022-03-17T20:51:47.7752113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:47.7763546Z 2022-03-17T20:51:47.7764178Z Running tests... 2022-03-17T20:51:47.7765369Z ---------------------------------------------------------------------- 2022-03-17T20:51:47.7782143Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:51:47.7783341Z 2022-03-17T20:51:47.7784439Z ---------------------------------------------------------------------- 2022-03-17T20:51:47.7785233Z Ran 1 test in 0.002s 2022-03-17T20:51:47.7785639Z 2022-03-17T20:51:47.7785883Z OK (skipped=1) 2022-03-17T20:51:47.7786698Z 2022-03-17T20:51:47.7786987Z Generating XML reports... 2022-03-17T20:51:47.7823064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205147.xml 2022-03-17T20:51:50.2496312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:50.2509681Z 2022-03-17T20:51:50.2509936Z Running tests... 2022-03-17T20:51:50.2510579Z ---------------------------------------------------------------------- 2022-03-17T20:51:50.2526629Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:51:50.2527067Z 2022-03-17T20:51:50.2528771Z ---------------------------------------------------------------------- 2022-03-17T20:51:50.2529250Z Ran 1 test in 0.002s 2022-03-17T20:51:50.2529463Z 2022-03-17T20:51:50.2529599Z OK (skipped=1) 2022-03-17T20:51:50.2529791Z 2022-03-17T20:51:50.2529907Z Generating XML reports... 2022-03-17T20:51:50.2566274Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205150.xml 2022-03-17T20:51:52.7334160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:52.7347094Z 2022-03-17T20:51:52.7347371Z Running tests... 2022-03-17T20:51:52.7347914Z ---------------------------------------------------------------------- 2022-03-17T20:51:52.7367585Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:51:52.7368140Z 2022-03-17T20:51:52.7368809Z ---------------------------------------------------------------------- 2022-03-17T20:51:52.7369169Z Ran 1 test in 0.002s 2022-03-17T20:51:52.7369412Z 2022-03-17T20:51:52.7369556Z OK (skipped=1) 2022-03-17T20:51:52.7369719Z 2022-03-17T20:51:52.7407079Z Generating XML reports... 2022-03-17T20:51:52.7407760Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205152.xml 2022-03-17T20:51:55.3098017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:55.3111839Z 2022-03-17T20:51:55.3112371Z Running tests... 2022-03-17T20:51:55.3113546Z ---------------------------------------------------------------------- 2022-03-17T20:51:55.3134608Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:51:55.3135712Z 2022-03-17T20:51:55.3136496Z ---------------------------------------------------------------------- 2022-03-17T20:51:55.3137252Z Ran 1 test in 0.002s 2022-03-17T20:51:55.3137616Z 2022-03-17T20:51:55.3137838Z OK (skipped=1) 2022-03-17T20:51:55.3138370Z 2022-03-17T20:51:55.3138740Z Generating XML reports... 2022-03-17T20:51:55.3176292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205155.xml 2022-03-17T20:51:57.9964199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:57.9979385Z 2022-03-17T20:51:57.9979655Z Running tests... 2022-03-17T20:51:57.9980199Z ---------------------------------------------------------------------- 2022-03-17T20:51:58.0000484Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:51:58.0000921Z 2022-03-17T20:51:58.0001268Z ---------------------------------------------------------------------- 2022-03-17T20:51:58.0001615Z Ran 1 test in 0.002s 2022-03-17T20:51:58.0001776Z 2022-03-17T20:51:58.0001965Z OK (skipped=1) 2022-03-17T20:51:58.0002120Z 2022-03-17T20:51:58.0002244Z Generating XML reports... 2022-03-17T20:51:58.0042020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205157.xml 2022-03-17T20:52:00.6889646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:00.6901257Z 2022-03-17T20:52:00.6901618Z Running tests... 2022-03-17T20:52:00.6902129Z ---------------------------------------------------------------------- 2022-03-17T20:52:00.6921150Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:00.6921666Z 2022-03-17T20:52:00.6922596Z ---------------------------------------------------------------------- 2022-03-17T20:52:00.6922963Z Ran 1 test in 0.002s 2022-03-17T20:52:00.6923142Z 2022-03-17T20:52:00.6923247Z OK (skipped=1) 2022-03-17T20:52:00.6923392Z 2022-03-17T20:52:00.6923516Z Generating XML reports... 2022-03-17T20:52:00.6962424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205200.xml 2022-03-17T20:52:03.1544015Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:03.1556045Z 2022-03-17T20:52:03.1556371Z Running tests... 2022-03-17T20:52:03.1557534Z ---------------------------------------------------------------------- 2022-03-17T20:52:03.1576884Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:03.1577804Z 2022-03-17T20:52:03.1578634Z ---------------------------------------------------------------------- 2022-03-17T20:52:03.1579410Z Ran 1 test in 0.002s 2022-03-17T20:52:03.1579780Z 2022-03-17T20:52:03.1579995Z OK (skipped=1) 2022-03-17T20:52:03.1580340Z 2022-03-17T20:52:03.1580611Z Generating XML reports... 2022-03-17T20:52:03.1617416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205203.xml 2022-03-17T20:52:05.6483819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:05.6494576Z 2022-03-17T20:52:05.6494909Z Running tests... 2022-03-17T20:52:05.6496070Z ---------------------------------------------------------------------- 2022-03-17T20:52:05.6515979Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:05.6516936Z 2022-03-17T20:52:05.6518260Z ---------------------------------------------------------------------- 2022-03-17T20:52:05.6519138Z Ran 1 test in 0.002s 2022-03-17T20:52:05.6519517Z 2022-03-17T20:52:05.6519777Z OK (skipped=1) 2022-03-17T20:52:05.6520165Z 2022-03-17T20:52:05.6520468Z Generating XML reports... 2022-03-17T20:52:05.6557479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205205.xml 2022-03-17T20:52:08.1477900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:08.1486727Z 2022-03-17T20:52:08.1486999Z Running tests... 2022-03-17T20:52:08.1488137Z ---------------------------------------------------------------------- 2022-03-17T20:52:08.1509299Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:08.1510302Z 2022-03-17T20:52:08.1511383Z ---------------------------------------------------------------------- 2022-03-17T20:52:08.1512218Z Ran 1 test in 0.002s 2022-03-17T20:52:08.1512589Z 2022-03-17T20:52:08.1512806Z OK (skipped=1) 2022-03-17T20:52:08.1513167Z 2022-03-17T20:52:08.1513435Z Generating XML reports... 2022-03-17T20:52:08.1550217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205208.xml 2022-03-17T20:52:10.6313175Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:10.6329371Z 2022-03-17T20:52:10.6329676Z Running tests... 2022-03-17T20:52:10.6330198Z ---------------------------------------------------------------------- 2022-03-17T20:52:10.6346038Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:10.6346523Z 2022-03-17T20:52:10.6347043Z ---------------------------------------------------------------------- 2022-03-17T20:52:10.6347405Z Ran 1 test in 0.002s 2022-03-17T20:52:10.6347573Z 2022-03-17T20:52:10.6347675Z OK (skipped=1) 2022-03-17T20:52:10.6347834Z 2022-03-17T20:52:10.6347953Z Generating XML reports... 2022-03-17T20:52:10.6385344Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205210.xml 2022-03-17T20:52:13.3213613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:13.3227258Z 2022-03-17T20:52:13.3228026Z Running tests... 2022-03-17T20:52:13.3229218Z ---------------------------------------------------------------------- 2022-03-17T20:52:13.3243516Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:13.3244359Z 2022-03-17T20:52:13.3245163Z ---------------------------------------------------------------------- 2022-03-17T20:52:13.3245937Z Ran 1 test in 0.002s 2022-03-17T20:52:13.3246312Z 2022-03-17T20:52:13.3246545Z OK (skipped=1) 2022-03-17T20:52:13.3246916Z 2022-03-17T20:52:13.3247199Z Generating XML reports... 2022-03-17T20:52:13.3286611Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205213.xml 2022-03-17T20:52:15.8075004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:15.8087673Z 2022-03-17T20:52:15.8087982Z Running tests... 2022-03-17T20:52:15.8088513Z ---------------------------------------------------------------------- 2022-03-17T20:52:15.8106139Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:15.8106610Z 2022-03-17T20:52:15.8107296Z ---------------------------------------------------------------------- 2022-03-17T20:52:15.8107734Z Ran 1 test in 0.002s 2022-03-17T20:52:15.8107935Z 2022-03-17T20:52:15.8108070Z OK (skipped=1) 2022-03-17T20:52:15.8108282Z 2022-03-17T20:52:15.8108431Z Generating XML reports... 2022-03-17T20:52:15.8145313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205215.xml 2022-03-17T20:52:18.3352274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:18.3363848Z 2022-03-17T20:52:18.3364332Z Running tests... 2022-03-17T20:52:18.3365658Z ---------------------------------------------------------------------- 2022-03-17T20:52:18.9322778Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6225 2022-03-17T20:52:18.9360108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6226 2022-03-17T20:52:18.9361692Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:52:20.0665692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:20.0678586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:20.0680825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:20.0766225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:20.0769525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:20.1690400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:20.5479789Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:52:20.5956383Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:52:29.8796873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:29.9562537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:29.9724572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:29.9731138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:29.9883991Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:52:29.9888757Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:52:30.0336243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0345134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0557062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0558792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0658803Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:52:30.0661313Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:52:30.0665039Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:52:30.0667426Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:52:30.0710509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0712056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0886571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.0888133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:32.7695420Z ok (14.433s) 2022-03-17T20:52:32.7696012Z 2022-03-17T20:52:32.7697327Z ---------------------------------------------------------------------- 2022-03-17T20:52:32.7699004Z Ran 1 test in 14.433s 2022-03-17T20:52:32.7699458Z 2022-03-17T20:52:32.7699671Z OK 2022-03-17T20:52:32.7701169Z 2022-03-17T20:52:32.7701608Z Generating XML reports... 2022-03-17T20:52:32.7781205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205218.xml 2022-03-17T20:52:35.4012893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:35.4029404Z 2022-03-17T20:52:35.4029709Z Running tests... 2022-03-17T20:52:35.4030138Z ---------------------------------------------------------------------- 2022-03-17T20:52:36.0025533Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6444 2022-03-17T20:52:36.0056269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6445 2022-03-17T20:52:36.0057630Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:52:37.0682401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:37.1366991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:37.1367806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:37.1389367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:37.1392564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:37.2379432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:37.6208325Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:52:37.6704631Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:52:49.8306393Z ok (14.427s) 2022-03-17T20:52:49.8306839Z 2022-03-17T20:52:49.8308041Z ---------------------------------------------------------------------- 2022-03-17T20:52:49.8308995Z Ran 1 test in 14.428s 2022-03-17T20:52:49.8309435Z 2022-03-17T20:52:49.8309603Z OK 2022-03-17T20:52:49.8309888Z 2022-03-17T20:52:49.8310161Z Generating XML reports... 2022-03-17T20:52:49.8383155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205235.xml 2022-03-17T20:52:52.5185084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:52:52.5198751Z 2022-03-17T20:52:52.5199067Z Running tests... 2022-03-17T20:52:52.5199592Z ---------------------------------------------------------------------- 2022-03-17T20:52:53.1190038Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6659 2022-03-17T20:52:53.1223251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6660 2022-03-17T20:52:53.1224536Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:52:54.2279503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:54.2535419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:54.2536408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:54.2584883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:54.2593385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:54.3550701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:03.6666792Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:53:03.7658302Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:53:03.9186697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:03.9188029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:06.6439433Z ok (14.124s) 2022-03-17T20:53:06.6439869Z 2022-03-17T20:53:06.6440830Z ---------------------------------------------------------------------- 2022-03-17T20:53:06.6441606Z Ran 1 test in 14.124s 2022-03-17T20:53:06.6442165Z 2022-03-17T20:53:06.6442349Z OK 2022-03-17T20:53:06.6442655Z 2022-03-17T20:53:06.6443697Z Generating XML reports... 2022-03-17T20:53:06.6523681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205252.xml 2022-03-17T20:53:09.2457586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:53:09.2470791Z 2022-03-17T20:53:09.2471040Z Running tests... 2022-03-17T20:53:09.2471495Z ---------------------------------------------------------------------- 2022-03-17T20:53:09.2551762Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.008s) 2022-03-17T20:53:09.2551998Z 2022-03-17T20:53:09.2552330Z ---------------------------------------------------------------------- 2022-03-17T20:53:09.2552633Z Ran 1 test in 0.008s 2022-03-17T20:53:09.2552796Z 2022-03-17T20:53:09.2552946Z OK (skipped=1) 2022-03-17T20:53:09.2553325Z 2022-03-17T20:53:09.2553603Z Generating XML reports... 2022-03-17T20:53:09.2594379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205309.xml 2022-03-17T20:53:11.8653309Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:53:11.8671333Z 2022-03-17T20:53:11.8672087Z Running tests... 2022-03-17T20:53:11.8673247Z ---------------------------------------------------------------------- 2022-03-17T20:53:12.4949410Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6945 2022-03-17T20:53:12.4987188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6946 2022-03-17T20:53:12.4988899Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:53:13.5716967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:13.6383664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:13.6385581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:13.6427742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:13.6429513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:13.7394338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:14.1192236Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:53:14.1716767Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:53:23.5399448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:23.5579798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:26.3308801Z ok (14.464s) 2022-03-17T20:53:26.3309221Z 2022-03-17T20:53:26.3310452Z ---------------------------------------------------------------------- 2022-03-17T20:53:26.3311969Z Ran 1 test in 14.464s 2022-03-17T20:53:26.3312376Z 2022-03-17T20:53:26.3312605Z OK 2022-03-17T20:53:26.3312994Z 2022-03-17T20:53:26.3313328Z Generating XML reports... 2022-03-17T20:53:26.3388314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205311.xml 2022-03-17T20:53:29.0181708Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:53:29.0194097Z 2022-03-17T20:53:29.0194422Z Running tests... 2022-03-17T20:53:29.0194940Z ---------------------------------------------------------------------- 2022-03-17T20:53:29.6458838Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7164 2022-03-17T20:53:29.6490656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7165 2022-03-17T20:53:29.6492223Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:53:30.7337354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:30.7492590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:30.7494657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:30.7541344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:30.7542910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:30.8506799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:32.6580860Z skip (3.638s) 2022-03-17T20:53:32.6581260Z 2022-03-17T20:53:32.6582150Z ---------------------------------------------------------------------- 2022-03-17T20:53:32.6582985Z Ran 1 test in 3.638s 2022-03-17T20:53:32.6583340Z 2022-03-17T20:53:32.6583570Z OK (skipped=1) 2022-03-17T20:53:32.6583916Z 2022-03-17T20:53:32.6584188Z Generating XML reports... 2022-03-17T20:53:32.6664818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205329.xml 2022-03-17T20:53:35.2696081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:53:35.2704900Z 2022-03-17T20:53:35.2705160Z Running tests... 2022-03-17T20:53:35.2706110Z ---------------------------------------------------------------------- 2022-03-17T20:53:35.8681093Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7372 2022-03-17T20:53:35.8718968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7373 2022-03-17T20:53:35.8720295Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:53:37.0126312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:37.0241812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:37.0245906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:37.0328825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:37.0334542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:37.1254988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:38.7794080Z skip (3.508s) 2022-03-17T20:53:38.7794470Z 2022-03-17T20:53:38.7796208Z ---------------------------------------------------------------------- 2022-03-17T20:53:38.7797154Z Ran 1 test in 3.509s 2022-03-17T20:53:38.7797524Z 2022-03-17T20:53:38.7797754Z OK (skipped=1) 2022-03-17T20:53:38.7798102Z 2022-03-17T20:53:38.7798349Z Generating XML reports... 2022-03-17T20:53:38.7882034Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205335.xml 2022-03-17T20:53:41.3764996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:53:41.3778592Z 2022-03-17T20:53:41.3778755Z Running tests... 2022-03-17T20:53:41.3779296Z ---------------------------------------------------------------------- 2022-03-17T20:53:42.0045564Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7580 2022-03-17T20:53:42.0082670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7581 2022-03-17T20:53:42.0084076Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:53:43.1091411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:43.1279068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:43.1281087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:43.1292892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:43.1295830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:43.2288029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:43.6073494Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:53:43.6722773Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:53:55.7331295Z ok (14.354s) 2022-03-17T20:53:55.7331897Z 2022-03-17T20:53:55.7333071Z ---------------------------------------------------------------------- 2022-03-17T20:53:55.7333918Z Ran 1 test in 14.355s 2022-03-17T20:53:55.7334292Z 2022-03-17T20:53:55.7334485Z OK 2022-03-17T20:53:55.7334764Z 2022-03-17T20:53:55.7335593Z Generating XML reports... 2022-03-17T20:53:55.7415880Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205341.xml 2022-03-17T20:53:58.3460425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:53:58.3475330Z 2022-03-17T20:53:58.3475692Z Running tests... 2022-03-17T20:53:58.3476248Z ---------------------------------------------------------------------- 2022-03-17T20:53:58.9470333Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7807 2022-03-17T20:53:58.9508881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7808 2022-03-17T20:53:58.9510515Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:54:00.0707999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:00.0730838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:00.0731912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:00.0808582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:00.0812733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:00.1740071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:00.5545252Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:00.6081781Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:12.8783556Z ok (14.531s) 2022-03-17T20:54:12.8784164Z 2022-03-17T20:54:12.8785284Z ---------------------------------------------------------------------- 2022-03-17T20:54:12.8786098Z Ran 1 test in 14.531s 2022-03-17T20:54:12.8786466Z 2022-03-17T20:54:12.8786656Z OK 2022-03-17T20:54:12.8786950Z 2022-03-17T20:54:12.8787215Z Generating XML reports... 2022-03-17T20:54:12.8865171Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205358.xml 2022-03-17T20:54:15.4109992Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:54:15.4122392Z 2022-03-17T20:54:15.4123134Z Running tests... 2022-03-17T20:54:15.4124455Z ---------------------------------------------------------------------- 2022-03-17T20:54:16.0119876Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8034 2022-03-17T20:54:16.0160013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8035 2022-03-17T20:54:16.0161483Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:54:17.1526784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:17.1670905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:17.1672858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:17.1730536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:17.1739485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:17.2689074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:17.6577950Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:17.7167064Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:29.6427635Z ok (14.230s) 2022-03-17T20:54:29.6428145Z 2022-03-17T20:54:29.6429378Z ---------------------------------------------------------------------- 2022-03-17T20:54:29.6430365Z Ran 1 test in 14.230s 2022-03-17T20:54:29.6430862Z 2022-03-17T20:54:29.6431192Z OK 2022-03-17T20:54:29.6431524Z 2022-03-17T20:54:29.6431817Z Generating XML reports... 2022-03-17T20:54:29.6510161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205415.xml 2022-03-17T20:54:32.3226633Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:54:32.3239889Z 2022-03-17T20:54:32.3240245Z Running tests... 2022-03-17T20:54:32.3240793Z ---------------------------------------------------------------------- 2022-03-17T20:54:32.9202404Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8249 2022-03-17T20:54:32.9241679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8250 2022-03-17T20:54:32.9244008Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:54:34.0410652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:34.0602344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:34.0603478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:34.0608753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:34.0619082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:34.1617148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:34.5431306Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:34.6061665Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:43.6750452Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:54:43.7724716Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:54:43.9326457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:54:43.9337762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:54:46.6547835Z ok (14.330s) 2022-03-17T20:54:46.6548275Z 2022-03-17T20:54:46.6551711Z ---------------------------------------------------------------------- 2022-03-17T20:54:46.6552160Z Ran 1 test in 14.331s 2022-03-17T20:54:46.6552424Z 2022-03-17T20:54:46.6552547Z OK 2022-03-17T20:54:46.6552721Z 2022-03-17T20:54:46.6553257Z Generating XML reports... 2022-03-17T20:54:46.6623342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205432.xml 2022-03-17T20:54:49.2806833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:54:49.2821036Z 2022-03-17T20:54:49.2821794Z Running tests... 2022-03-17T20:54:49.2823030Z ---------------------------------------------------------------------- 2022-03-17T20:54:49.8866837Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8468 2022-03-17T20:54:49.8924995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8469 2022-03-17T20:54:49.8926302Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:54:51.0401286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:51.0418643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:51.0420812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:51.0502429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:51.0503815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:51.1429927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:51.4275938Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:51.5197165Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:54:51.6121614Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:54:51.6122640Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:54:51.6271849Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:54:51.6274266Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:55:03.5256046Z ok (14.243s) 2022-03-17T20:55:03.5256622Z 2022-03-17T20:55:03.5257935Z ---------------------------------------------------------------------- 2022-03-17T20:55:03.5258711Z Ran 1 test in 14.243s 2022-03-17T20:55:03.5259080Z 2022-03-17T20:55:03.5259318Z OK 2022-03-17T20:55:03.5259768Z 2022-03-17T20:55:03.5260129Z Generating XML reports... 2022-03-17T20:55:03.5340351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205449.xml 2022-03-17T20:55:06.0812824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:55:06.0824831Z 2022-03-17T20:55:06.0825778Z Running tests... 2022-03-17T20:55:06.0827136Z ---------------------------------------------------------------------- 2022-03-17T20:55:06.6832685Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8687 2022-03-17T20:55:06.6872821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8688 2022-03-17T20:55:06.6874508Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:55:07.8255675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:07.8390966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:07.8393517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:07.8459379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:07.8465927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:07.9410397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:08.3255123Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:08.3849501Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:17.6167902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:17.6170122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:17.6668582Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:55:17.6673359Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:55:20.2198070Z ok (14.137s) 2022-03-17T20:55:20.2198630Z 2022-03-17T20:55:20.2199810Z ---------------------------------------------------------------------- 2022-03-17T20:55:20.2200825Z Ran 1 test in 14.137s 2022-03-17T20:55:20.2201285Z 2022-03-17T20:55:20.2201530Z OK 2022-03-17T20:55:20.2202282Z 2022-03-17T20:55:20.2202608Z Generating XML reports... 2022-03-17T20:55:20.2283528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205506.xml 2022-03-17T20:55:22.7770371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:55:22.7783080Z 2022-03-17T20:55:22.7783326Z Running tests... 2022-03-17T20:55:22.7783871Z ---------------------------------------------------------------------- 2022-03-17T20:55:23.3659272Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.587s) 2022-03-17T20:55:23.3659596Z 2022-03-17T20:55:23.3660067Z ---------------------------------------------------------------------- 2022-03-17T20:55:23.3660604Z Ran 1 test in 0.588s 2022-03-17T20:55:23.3660798Z 2022-03-17T20:55:23.3660953Z OK (skipped=1) 2022-03-17T20:55:23.3661134Z 2022-03-17T20:55:23.3661259Z Generating XML reports... 2022-03-17T20:55:23.3688253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205522.xml 2022-03-17T20:55:25.9280185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:55:25.9294107Z 2022-03-17T20:55:25.9294693Z Running tests... 2022-03-17T20:55:25.9295834Z ---------------------------------------------------------------------- 2022-03-17T20:55:26.5273418Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8979 2022-03-17T20:55:26.5310697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8980 2022-03-17T20:55:26.5312075Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:55:27.6403371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:27.6574660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:27.6576613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:27.6609565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:27.6619830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:27.7581480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:28.0520323Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:28.1357369Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:29.8406411Z ok (3.911s) 2022-03-17T20:55:29.8406793Z 2022-03-17T20:55:29.8407711Z ---------------------------------------------------------------------- 2022-03-17T20:55:29.8408514Z Ran 1 test in 3.911s 2022-03-17T20:55:29.8408886Z 2022-03-17T20:55:29.8409085Z OK 2022-03-17T20:55:29.8409363Z 2022-03-17T20:55:29.8409660Z Generating XML reports... 2022-03-17T20:55:29.8486578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205525.xml 2022-03-17T20:55:32.4863461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:55:32.4871922Z 2022-03-17T20:55:32.4872238Z Running tests... 2022-03-17T20:55:32.4873334Z ---------------------------------------------------------------------- 2022-03-17T20:55:33.0877773Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9194 2022-03-17T20:55:33.0911186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9195 2022-03-17T20:55:33.0912468Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:55:34.1600265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:34.2409274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:34.2410334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:34.2509731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:34.2511108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:34.3425219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:34.7412508Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:34.7861893Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:34.8821234Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:55:34.8823497Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:55:34.8852149Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:55:34.8854361Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:55:43.9926200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:43.9934930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:44.0009892Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:55:44.0014859Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:55:44.0166983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:44.0168273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:44.0254481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:44.0255769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:55:46.6214935Z ok (14.134s) 2022-03-17T20:55:46.6215340Z 2022-03-17T20:55:46.6216337Z ---------------------------------------------------------------------- 2022-03-17T20:55:46.6217130Z Ran 1 test in 14.134s 2022-03-17T20:55:46.6217536Z 2022-03-17T20:55:46.6222952Z OK 2022-03-17T20:55:46.6223338Z 2022-03-17T20:55:46.6223576Z Generating XML reports... 2022-03-17T20:55:46.6293762Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205532.xml 2022-03-17T20:55:49.1612341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:55:49.1624842Z 2022-03-17T20:55:49.1625121Z Running tests... 2022-03-17T20:55:49.1625716Z ---------------------------------------------------------------------- 2022-03-17T20:55:49.7617076Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9413 2022-03-17T20:55:49.7646929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9414 2022-03-17T20:55:49.7648395Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:55:50.8486358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:50.8908654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:50.8910547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:50.8992511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:50.8994027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:50.9916661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:51.3680512Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:51.4191662Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:55:51.5239747Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:55:51.5242130Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:55:51.5245290Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:55:51.5247786Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:56:00.6378924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:00.6384622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:03.1973753Z ok (14.034s) 2022-03-17T20:56:03.1974203Z 2022-03-17T20:56:03.1975196Z ---------------------------------------------------------------------- 2022-03-17T20:56:03.1975983Z Ran 1 test in 14.034s 2022-03-17T20:56:03.1976347Z 2022-03-17T20:56:03.1976548Z OK 2022-03-17T20:56:03.1977195Z 2022-03-17T20:56:03.1977622Z Generating XML reports... 2022-03-17T20:56:03.2045526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205549.xml 2022-03-17T20:56:05.7473678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:56:05.7484880Z 2022-03-17T20:56:05.7485232Z Running tests... 2022-03-17T20:56:05.7485656Z ---------------------------------------------------------------------- 2022-03-17T20:56:06.3528057Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9632 2022-03-17T20:56:06.3558466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9633 2022-03-17T20:56:06.3559592Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:56:07.4869367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:07.4942593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:07.4944522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:07.4970696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:07.4972044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:07.5957603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:07.9907832Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:08.0379950Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:17.4029517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:17.4065059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:17.4865082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:17.4866370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:20.2849000Z ok (14.536s) 2022-03-17T20:56:20.2849390Z 2022-03-17T20:56:20.2850376Z ---------------------------------------------------------------------- 2022-03-17T20:56:20.2851187Z Ran 1 test in 14.536s 2022-03-17T20:56:20.2851575Z 2022-03-17T20:56:20.2851753Z OK 2022-03-17T20:56:20.2852087Z 2022-03-17T20:56:20.2852371Z Generating XML reports... 2022-03-17T20:56:20.2936452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205605.xml 2022-03-17T20:56:22.9006278Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:56:22.9018286Z 2022-03-17T20:56:22.9019170Z Running tests... 2022-03-17T20:56:22.9019983Z ---------------------------------------------------------------------- 2022-03-17T20:56:23.4929504Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... skip (0.591s) 2022-03-17T20:56:23.4930228Z 2022-03-17T20:56:23.4931074Z ---------------------------------------------------------------------- 2022-03-17T20:56:23.4931871Z Ran 1 test in 0.591s 2022-03-17T20:56:23.4932258Z 2022-03-17T20:56:23.4932500Z OK (skipped=1) 2022-03-17T20:56:23.4932852Z 2022-03-17T20:56:23.4933146Z Generating XML reports... 2022-03-17T20:56:23.4958867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205622.xml 2022-03-17T20:56:26.0738190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:56:26.0751981Z 2022-03-17T20:56:26.0752499Z Running tests... 2022-03-17T20:56:26.0753661Z ---------------------------------------------------------------------- 2022-03-17T20:56:26.7104162Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9918 2022-03-17T20:56:26.7147501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9919 2022-03-17T20:56:26.7148974Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:56:27.8431196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:27.8432454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:27.8434427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:27.8530893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:27.8533851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:27.9443353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:28.3264437Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:28.3742067Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:30.3259759Z ok (4.250s) 2022-03-17T20:56:30.3260620Z 2022-03-17T20:56:30.3261527Z ---------------------------------------------------------------------- 2022-03-17T20:56:30.3262848Z Ran 1 test in 4.251s 2022-03-17T20:56:30.3263243Z 2022-03-17T20:56:30.3263431Z OK 2022-03-17T20:56:30.3263709Z 2022-03-17T20:56:30.3265598Z Generating XML reports... 2022-03-17T20:56:30.3343121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205626.xml 2022-03-17T20:56:32.9045850Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:56:32.9058390Z 2022-03-17T20:56:32.9059056Z Running tests... 2022-03-17T20:56:32.9060231Z ---------------------------------------------------------------------- 2022-03-17T20:56:33.5052795Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10133 2022-03-17T20:56:33.5096333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10134 2022-03-17T20:56:33.5097718Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:56:34.6294912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:34.6429296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:34.6430039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:34.6499125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:34.6504741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:34.7441215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:35.0331916Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:35.1285174Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:36.8187671Z ok (3.912s) 2022-03-17T20:56:36.8188090Z 2022-03-17T20:56:36.8189099Z ---------------------------------------------------------------------- 2022-03-17T20:56:36.8190216Z Ran 1 test in 3.913s 2022-03-17T20:56:36.8190613Z 2022-03-17T20:56:36.8190800Z OK 2022-03-17T20:56:36.8191077Z 2022-03-17T20:56:36.8191358Z Generating XML reports... 2022-03-17T20:56:36.8263925Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205632.xml 2022-03-17T20:56:39.3974464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:56:39.3988104Z 2022-03-17T20:56:39.3988714Z Running tests... 2022-03-17T20:56:39.3989862Z ---------------------------------------------------------------------- 2022-03-17T20:56:40.0351837Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10348 2022-03-17T20:56:40.0386817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10349 2022-03-17T20:56:40.0388139Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:56:41.1313863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:41.1674737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:41.1676637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:41.1721634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:41.1729194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:41.2686203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:41.6519846Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:41.7093223Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:50.8586339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:50.8614252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:56:53.4602990Z ok (14.061s) 2022-03-17T20:56:53.4603393Z 2022-03-17T20:56:53.4604341Z ---------------------------------------------------------------------- 2022-03-17T20:56:53.4605106Z Ran 1 test in 14.061s 2022-03-17T20:56:53.4605471Z 2022-03-17T20:56:53.4607184Z OK 2022-03-17T20:56:53.4607494Z 2022-03-17T20:56:53.4607770Z Generating XML reports... 2022-03-17T20:56:53.4688380Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205639.xml 2022-03-17T20:56:56.0858955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:56:56.0872467Z 2022-03-17T20:56:56.0873065Z Running tests... 2022-03-17T20:56:56.0874289Z ---------------------------------------------------------------------- 2022-03-17T20:56:56.6916554Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10567 2022-03-17T20:56:56.6948400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10568 2022-03-17T20:56:56.6949761Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:56:57.7890717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:57.7947199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:57.7948854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:57.7990525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:57.7995953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:57.8963392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:58.1448701Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:56:58.2533697Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:56:58.2535263Z Running test: batch_norm_net sync interval 2022-03-17T20:56:58.2536058Z 1 with iteration mapping 2022-03-17T20:56:58.2536846Z {0: 0, 1: 2} 2022-03-17T20:56:58.3026157Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:56:58.3574767Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:57:07.5845016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:07.5924426Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:07.5926699Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:07.5929759Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:07.5931839Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:07.7350984Z Running test: batch_norm_net sync interval 2022-03-17T20:57:07.7351791Z 1 with iteration mapping 2022-03-17T20:57:07.7352410Z {0: 0, 1: 3} 2022-03-17T20:57:07.7412623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:07.7481695Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:07.7484161Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:07.7487242Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:07.7489780Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:07.7551321Z Running test: batch_norm_net sync interval 2022-03-17T20:57:07.7552177Z 1 with iteration mapping 2022-03-17T20:57:07.7552829Z {0: 0, 1: 10} 2022-03-17T20:57:07.7608268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:07.7910065Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:07.7912310Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:07.7915365Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:07.7917490Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:07.7979758Z Running test: batch_norm_net sync interval 2022-03-17T20:57:07.7980629Z 1 with iteration mapping 2022-03-17T20:57:07.7981292Z {0: 5, 1: 7} 2022-03-17T20:57:16.7885161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:16.7889105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:16.8091263Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:16.8094002Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:16.8096921Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:16.8098894Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:16.8188706Z Running test: batch_norm_net sync interval 2022-03-17T20:57:16.8189475Z 1 with iteration mapping 2022-03-17T20:57:16.8190130Z {0: 5, 1: 8} 2022-03-17T20:57:16.8245897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:16.8247130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:16.8448776Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:16.8450831Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:16.8453696Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:16.8456115Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:16.8530348Z Running test: batch_norm_net sync interval 2022-03-17T20:57:16.8531118Z 1 with iteration mapping 2022-03-17T20:57:16.8531758Z {0: 5, 1: 15} 2022-03-17T20:57:16.8575028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:16.8576267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:16.8940191Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:16.8942287Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:16.8945174Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:16.8947185Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:16.9035379Z Running test: large_conv_model sync interval 2022-03-17T20:57:16.9036161Z 1 with iteration mapping 2022-03-17T20:57:16.9037073Z {0: 0, 1: 2} 2022-03-17T20:57:17.2267979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:17.4123059Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:17.4125462Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:17.4128433Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:17.4130451Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:17.4178536Z Running test: large_conv_model sync interval 2022-03-17T20:57:17.4179508Z 1 with iteration mapping 2022-03-17T20:57:17.4180395Z {0: 0, 1: 3} 2022-03-17T20:57:17.6030750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:17.9697237Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:17.9699324Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:17.9702239Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:17.9704724Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:17.9754207Z Running test: large_conv_model sync interval 2022-03-17T20:57:17.9755163Z 1 with iteration mapping 2022-03-17T20:57:17.9755963Z {0: 0, 1: 10} 2022-03-17T20:57:18.1599479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:19.8168561Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:19.8170630Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:19.8173512Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:19.8175475Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:19.8222087Z Running test: large_conv_model sync interval 2022-03-17T20:57:19.8223098Z 1 with iteration mapping 2022-03-17T20:57:19.8223878Z {0: 5, 1: 7} 2022-03-17T20:57:20.1370022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:20.1371528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:21.2324349Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:21.2326706Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:21.2329650Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:21.2331610Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:21.2380506Z Running test: large_conv_model sync interval 2022-03-17T20:57:21.2381454Z 1 with iteration mapping 2022-03-17T20:57:21.2382224Z {0: 5, 1: 8} 2022-03-17T20:57:21.4225796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:21.4227336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:22.7017738Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:22.7019834Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:22.7022738Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:22.7025230Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:22.7078028Z Running test: large_conv_model sync interval 2022-03-17T20:57:22.7078985Z 1 with iteration mapping 2022-03-17T20:57:22.7079798Z {0: 5, 1: 15} 2022-03-17T20:57:22.8929595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:22.8931081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.4493173Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.4495312Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.4498314Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.4500277Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.4557586Z Running test: small_model sync interval 2022-03-17T20:57:25.4558550Z 1 with iteration mapping 2022-03-17T20:57:25.4559295Z {0: 0, 1: 2} 2022-03-17T20:57:25.4618102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.4672494Z Running test: small_model sync interval 2022-03-17T20:57:25.4673419Z 1 with iteration mapping 2022-03-17T20:57:25.4674228Z {0: 0, 1: 3} 2022-03-17T20:57:25.4708504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.4779785Z Running test: small_model sync interval 2022-03-17T20:57:25.4781135Z 1 with iteration mapping 2022-03-17T20:57:25.4781823Z {0: 0, 1: 10} 2022-03-17T20:57:25.4818357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5013965Z Running test: small_model sync interval 2022-03-17T20:57:25.5014897Z 1 with iteration mapping 2022-03-17T20:57:25.5015718Z {0: 5, 1: 7} 2022-03-17T20:57:25.5061968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5063528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5214433Z Running test: small_model sync interval 2022-03-17T20:57:25.5215376Z 1 with iteration mapping 2022-03-17T20:57:25.5216149Z {0: 5, 1: 8} 2022-03-17T20:57:25.5250670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5252177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5411493Z Running test: small_model sync interval 2022-03-17T20:57:25.5412417Z 1 with iteration mapping 2022-03-17T20:57:25.5413246Z {0: 5, 1: 15} 2022-03-17T20:57:25.5449637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5451135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.5730405Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.5731479Z 1 with iteration mapping 2022-03-17T20:57:25.5732208Z {0: 0, 1: 2} 2022-03-17T20:57:25.5792334Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.5794388Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.5797259Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.5799212Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.5813851Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.5814969Z 1 with iteration mapping 2022-03-17T20:57:25.5815690Z {0: 0, 1: 3} 2022-03-17T20:57:25.5884784Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.5886921Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.5889830Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.5891809Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.5904800Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.5906201Z 1 with iteration mapping 2022-03-17T20:57:25.5906876Z {0: 0, 1: 10} 2022-03-17T20:57:25.6085725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6087822Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6090734Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6092727Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6118560Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.6119617Z 1 with iteration mapping 2022-03-17T20:57:25.6120349Z {0: 5, 1: 7} 2022-03-17T20:57:25.6142648Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:57:25.6269981Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6272058Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6274998Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6276999Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6302541Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.6303675Z 1 with iteration mapping 2022-03-17T20:57:25.6304386Z {0: 5, 1: 8} 2022-03-17T20:57:25.6471079Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6473170Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6476100Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6478092Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6514690Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.6515785Z 1 with iteration mapping 2022-03-17T20:57:25.6516419Z {0: 5, 1: 15} 2022-03-17T20:57:25.6791326Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6793458Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6796369Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6798318Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6823106Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.6824167Z 1 with iteration mapping 2022-03-17T20:57:25.6824892Z {0: 0, 1: 2} 2022-03-17T20:57:25.6846428Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:57:25.6880056Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6882586Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6885519Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6887519Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6901907Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.6902655Z 1 with iteration mapping 2022-03-17T20:57:25.6903200Z {0: 0, 1: 3} 2022-03-17T20:57:25.6973616Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6975072Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6977224Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.6978647Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.6997836Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.6998943Z 1 with iteration mapping 2022-03-17T20:57:25.6999585Z {0: 0, 1: 10} 2022-03-17T20:57:25.7190827Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7192960Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7195875Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7197861Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7222943Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.7224021Z 1 with iteration mapping 2022-03-17T20:57:25.7224717Z {0: 5, 1: 7} 2022-03-17T20:57:25.7366695Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7369095Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7372110Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7374064Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7398936Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.7399981Z 1 with iteration mapping 2022-03-17T20:57:25.7400733Z {0: 5, 1: 8} 2022-03-17T20:57:25.7566798Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7568897Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7571740Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7573697Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7599275Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:25.7600333Z 1 with iteration mapping 2022-03-17T20:57:25.7601076Z {0: 5, 1: 15} 2022-03-17T20:57:25.7879261Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7881346Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7884576Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:25.7886570Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:25.7911165Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:25.7912155Z 1 with iteration mapping 2022-03-17T20:57:25.7912931Z {0: 0, 1: 2} 2022-03-17T20:57:25.7931615Z registered hook 2022-03-17T20:57:25.7945106Z registered hook 2022-03-17T20:57:25.7996000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8064593Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:25.8065656Z 1 with iteration mapping 2022-03-17T20:57:25.8066536Z {0: 0, 1: 3} 2022-03-17T20:57:25.8075355Z registered hook 2022-03-17T20:57:25.8079895Z registered hook 2022-03-17T20:57:25.8116471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8193312Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:25.8194157Z 1 with iteration mapping 2022-03-17T20:57:25.8194773Z {0: 0, 1: 10} 2022-03-17T20:57:25.8201259Z registered hook 2022-03-17T20:57:25.8202929Z registered hook 2022-03-17T20:57:25.8235038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8445708Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:25.8446561Z 1 with iteration mapping 2022-03-17T20:57:25.8447165Z {0: 5, 1: 7} 2022-03-17T20:57:25.8458903Z registered hook 2022-03-17T20:57:25.8460660Z registered hook 2022-03-17T20:57:25.8495178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8496406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8655326Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:25.8656291Z 1 with iteration mapping 2022-03-17T20:57:25.8656919Z {0: 5, 1: 8} 2022-03-17T20:57:25.8665567Z registered hook 2022-03-17T20:57:25.8668789Z registered hook 2022-03-17T20:57:25.8701726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8702973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8875634Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:25.8876582Z 1 with iteration mapping 2022-03-17T20:57:25.8877208Z {0: 5, 1: 15} 2022-03-17T20:57:25.8891531Z registered hook 2022-03-17T20:57:25.8892487Z registered hook 2022-03-17T20:57:25.8921716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.8923400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.9247667Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:25.9248530Z 1 with iteration mapping 2022-03-17T20:57:25.9249134Z {0: 0, 1: 2} 2022-03-17T20:57:25.9261925Z registered hook 2022-03-17T20:57:25.9263632Z registered hook 2022-03-17T20:57:25.9274375Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:57:25.9284772Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:57:25.9303855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.9305427Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:57:25.9310913Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:57:25.9717820Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:25.9719011Z 1 with iteration mapping 2022-03-17T20:57:25.9719890Z {0: 0, 1: 3} 2022-03-17T20:57:25.9734439Z registered hook 2022-03-17T20:57:25.9800909Z registered hook 2022-03-17T20:57:25.9811287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:25.9911361Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:25.9911741Z 1 with iteration mapping 2022-03-17T20:57:25.9912019Z {0: 0, 1: 10} 2022-03-17T20:57:25.9921612Z registered hook 2022-03-17T20:57:25.9925566Z registered hook 2022-03-17T20:57:25.9960822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.0246957Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:26.0247767Z 1 with iteration mapping 2022-03-17T20:57:26.0248374Z {0: 5, 1: 7} 2022-03-17T20:57:26.0258298Z registered hook 2022-03-17T20:57:26.0259228Z registered hook 2022-03-17T20:57:26.0297270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.0298443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.0516524Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:26.0517390Z 1 with iteration mapping 2022-03-17T20:57:26.0517968Z {0: 5, 1: 8} 2022-03-17T20:57:26.0530392Z registered hook 2022-03-17T20:57:26.0531593Z registered hook 2022-03-17T20:57:26.0570591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.0572161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.0797125Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:26.0798169Z 1 with iteration mapping 2022-03-17T20:57:26.0799159Z {0: 5, 1: 15} 2022-03-17T20:57:26.0809752Z registered hook 2022-03-17T20:57:26.0810757Z registered hook 2022-03-17T20:57:26.0852116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.0853640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.1232975Z Running test: batch_norm_net sync interval 2022-03-17T20:57:26.1233905Z 2 with iteration mapping 2022-03-17T20:57:26.1234519Z {0: 0, 1: 2} 2022-03-17T20:57:26.1281987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.1318081Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.1320294Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.1323483Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.1325458Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.1381409Z Running test: batch_norm_net sync interval 2022-03-17T20:57:26.1382168Z 2 with iteration mapping 2022-03-17T20:57:26.1383110Z {0: 0, 1: 3} 2022-03-17T20:57:26.1432332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.1484612Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.1486678Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.1489518Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.1491515Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.1548508Z Running test: batch_norm_net sync interval 2022-03-17T20:57:26.1549355Z 2 with iteration mapping 2022-03-17T20:57:26.1550066Z {0: 0, 1: 10} 2022-03-17T20:57:26.1599847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.1786211Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.1788519Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.1791696Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.1793847Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.1849969Z Running test: batch_norm_net sync interval 2022-03-17T20:57:26.1851036Z 2 with iteration mapping 2022-03-17T20:57:26.1851655Z {0: 5, 1: 7} 2022-03-17T20:57:26.1898771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.1900006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.2031936Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.2034020Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.2036865Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.2038804Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.2113180Z Running test: batch_norm_net sync interval 2022-03-17T20:57:26.2113966Z 2 with iteration mapping 2022-03-17T20:57:26.2114587Z {0: 5, 1: 8} 2022-03-17T20:57:26.2166509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.2167919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.2320441Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.2322811Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.2325654Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.2327648Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.2386163Z Running test: batch_norm_net sync interval 2022-03-17T20:57:26.2386944Z 2 with iteration mapping 2022-03-17T20:57:26.2387559Z {0: 5, 1: 15} 2022-03-17T20:57:26.2435866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.2437125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.2719272Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.2721321Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.2724550Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.2726785Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.2781666Z Running test: large_conv_model sync interval 2022-03-17T20:57:26.2782539Z 3 with iteration mapping 2022-03-17T20:57:26.2783162Z {0: 0, 1: 2} 2022-03-17T20:57:26.4632668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:26.6467866Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.6470027Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.6473042Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:26.6475089Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:26.6527658Z Running test: large_conv_model sync interval 2022-03-17T20:57:26.6528626Z 3 with iteration mapping 2022-03-17T20:57:26.6529393Z {0: 0, 1: 3} 2022-03-17T20:57:26.8379949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:27.2034766Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:27.2036847Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:27.2039696Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:27.2041656Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:27.2094265Z Running test: large_conv_model sync interval 2022-03-17T20:57:27.2095258Z 3 with iteration mapping 2022-03-17T20:57:27.2096038Z {0: 0, 1: 10} 2022-03-17T20:57:27.3941333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:29.0354352Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:29.0356467Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:29.0359350Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:29.0361312Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:29.0412306Z Running test: large_conv_model sync interval 2022-03-17T20:57:29.0413298Z 3 with iteration mapping 2022-03-17T20:57:29.0414402Z {0: 5, 1: 7} 2022-03-17T20:57:29.2260163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:29.2261700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:30.3207983Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:30.3210092Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:30.3212974Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:30.3214974Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:30.3263412Z Running test: large_conv_model sync interval 2022-03-17T20:57:30.3264372Z 3 with iteration mapping 2022-03-17T20:57:30.3265151Z {0: 5, 1: 8} 2022-03-17T20:57:30.5110811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:30.5112892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:31.7879054Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:31.7881157Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:31.7884327Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:31.7886327Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:31.7934215Z Running test: large_conv_model sync interval 2022-03-17T20:57:31.7935182Z 3 with iteration mapping 2022-03-17T20:57:31.7935957Z {0: 5, 1: 15} 2022-03-17T20:57:31.9780180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:31.9781730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.5559705Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.5561800Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.5565073Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.5567046Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.5616000Z Running test: small_model sync interval 2022-03-17T20:57:34.5617031Z 4 with iteration mapping 2022-03-17T20:57:34.5617683Z {0: 0, 1: 2} 2022-03-17T20:57:34.5655145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.5698691Z Running test: small_model sync interval 2022-03-17T20:57:34.5699676Z 4 with iteration mapping 2022-03-17T20:57:34.5700485Z {0: 0, 1: 3} 2022-03-17T20:57:34.5732409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.5784750Z Running test: small_model sync interval 2022-03-17T20:57:34.5785755Z 4 with iteration mapping 2022-03-17T20:57:34.5786555Z {0: 0, 1: 10} 2022-03-17T20:57:34.5822317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.5952342Z Running test: small_model sync interval 2022-03-17T20:57:34.5953326Z 4 with iteration mapping 2022-03-17T20:57:34.5954139Z {0: 5, 1: 7} 2022-03-17T20:57:34.5992736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.5994313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.6085139Z Running test: small_model sync interval 2022-03-17T20:57:34.6086126Z 4 with iteration mapping 2022-03-17T20:57:34.6087289Z {0: 5, 1: 8} 2022-03-17T20:57:34.6123016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.6124521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.6223028Z Running test: small_model sync interval 2022-03-17T20:57:34.6224016Z 4 with iteration mapping 2022-03-17T20:57:34.6224861Z {0: 5, 1: 15} 2022-03-17T20:57:34.6257263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.6258798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.6434614Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.6435740Z 5 with iteration mapping 2022-03-17T20:57:34.6436509Z {0: 0, 1: 2} 2022-03-17T20:57:34.6490963Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6493054Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6495932Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6497886Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6509359Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.6510463Z 5 with iteration mapping 2022-03-17T20:57:34.6511196Z {0: 0, 1: 3} 2022-03-17T20:57:34.6583456Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6585801Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6588729Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6590733Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6615087Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.6616213Z 5 with iteration mapping 2022-03-17T20:57:34.6616959Z {0: 0, 1: 10} 2022-03-17T20:57:34.6733335Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6735437Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6738362Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6740666Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6749922Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.6751012Z 5 with iteration mapping 2022-03-17T20:57:34.6751766Z {0: 5, 1: 7} 2022-03-17T20:57:34.6836581Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6838676Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6841511Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6843627Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6854797Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.6855869Z 5 with iteration mapping 2022-03-17T20:57:34.6856601Z {0: 5, 1: 8} 2022-03-17T20:57:34.6949616Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6951707Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6954571Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.6956793Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.6971098Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.6972253Z 5 with iteration mapping 2022-03-17T20:57:34.6973027Z {0: 5, 1: 15} 2022-03-17T20:57:34.7123491Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7126355Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7130595Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7133573Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7154642Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.7155742Z 6 with iteration mapping 2022-03-17T20:57:34.7156470Z {0: 0, 1: 2} 2022-03-17T20:57:34.7209773Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7211934Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7214831Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7216780Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7227590Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.7228399Z 6 with iteration mapping 2022-03-17T20:57:34.7228946Z {0: 0, 1: 3} 2022-03-17T20:57:34.7278977Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7280479Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7282662Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7284099Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7297997Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.7298771Z 6 with iteration mapping 2022-03-17T20:57:34.7299337Z {0: 0, 1: 10} 2022-03-17T20:57:34.7406774Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7408282Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7410403Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7411834Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7425967Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.7426790Z 6 with iteration mapping 2022-03-17T20:57:34.7427363Z {0: 5, 1: 7} 2022-03-17T20:57:34.7514501Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7516011Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7518407Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7519839Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7532611Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.7533722Z 6 with iteration mapping 2022-03-17T20:57:34.7534519Z {0: 5, 1: 8} 2022-03-17T20:57:34.7630469Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7632571Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7635395Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7637381Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7651189Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:34.7652259Z 6 with iteration mapping 2022-03-17T20:57:34.7653007Z {0: 5, 1: 15} 2022-03-17T20:57:34.7803371Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7805456Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7808701Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:34.7810724Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:34.7821600Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:34.7822686Z 7 with iteration mapping 2022-03-17T20:57:34.7823463Z {0: 0, 1: 2} 2022-03-17T20:57:34.7858719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.7902719Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:34.7903801Z 7 with iteration mapping 2022-03-17T20:57:34.7904583Z {0: 0, 1: 3} 2022-03-17T20:57:34.7938951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.7991538Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:34.7992627Z 7 with iteration mapping 2022-03-17T20:57:34.7993459Z {0: 0, 1: 10} 2022-03-17T20:57:34.8031607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8149217Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:34.8150281Z 7 with iteration mapping 2022-03-17T20:57:34.8151563Z {0: 5, 1: 7} 2022-03-17T20:57:34.8186901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8188427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8262327Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:34.8263386Z 7 with iteration mapping 2022-03-17T20:57:34.8264172Z {0: 5, 1: 8} 2022-03-17T20:57:34.8297508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8299007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8393055Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:34.8394123Z 7 with iteration mapping 2022-03-17T20:57:34.8395030Z {0: 5, 1: 15} 2022-03-17T20:57:34.8427587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8429135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8582989Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:34.8584065Z 8 with iteration mapping 2022-03-17T20:57:34.8584845Z {0: 0, 1: 2} 2022-03-17T20:57:34.8617658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8660444Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:34.8661519Z 8 with iteration mapping 2022-03-17T20:57:34.8662292Z {0: 0, 1: 3} 2022-03-17T20:57:34.8697899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8749124Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:34.8750156Z 8 with iteration mapping 2022-03-17T20:57:34.8750985Z {0: 0, 1: 10} 2022-03-17T20:57:34.8784086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8894914Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:34.8895998Z 8 with iteration mapping 2022-03-17T20:57:34.8896839Z {0: 5, 1: 7} 2022-03-17T20:57:34.8939856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.8941330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9018418Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:34.9019512Z 8 with iteration mapping 2022-03-17T20:57:34.9020324Z {0: 5, 1: 8} 2022-03-17T20:57:34.9054792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9056343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9141220Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:34.9142247Z 8 with iteration mapping 2022-03-17T20:57:34.9143058Z {0: 5, 1: 15} 2022-03-17T20:57:34.9175902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9177479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9320433Z Running test: batch_norm_net sync interval 2022-03-17T20:57:34.9321421Z 1 with iteration mapping 2022-03-17T20:57:34.9322428Z {0: 0, 1: 2} 2022-03-17T20:57:34.9348781Z Running test: batch_norm_net sync interval 2022-03-17T20:57:34.9349798Z 1 with iteration mapping 2022-03-17T20:57:34.9350620Z {0: 0, 1: 3} 2022-03-17T20:57:34.9368890Z Running test: batch_norm_net sync interval 2022-03-17T20:57:34.9370250Z 1 with iteration mapping 2022-03-17T20:57:34.9370897Z {0: 0, 1: 10} 2022-03-17T20:57:34.9388812Z Running test: batch_norm_net sync interval 2022-03-17T20:57:34.9389798Z 1 with iteration mapping 2022-03-17T20:57:34.9390571Z {0: 5, 1: 7} 2022-03-17T20:57:34.9439243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9440779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9554512Z Running test: batch_norm_net sync interval 2022-03-17T20:57:34.9555515Z 1 with iteration mapping 2022-03-17T20:57:34.9556296Z {0: 5, 1: 8} 2022-03-17T20:57:34.9609456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9610993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9719673Z Running test: batch_norm_net sync interval 2022-03-17T20:57:34.9720650Z 1 with iteration mapping 2022-03-17T20:57:34.9721420Z {0: 5, 1: 15} 2022-03-17T20:57:34.9781014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9782571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:34.9885048Z Running test: large_conv_model sync interval 2022-03-17T20:57:34.9886107Z 1 with iteration mapping 2022-03-17T20:57:34.9886911Z {0: 0, 1: 2} 2022-03-17T20:57:34.9915888Z Running test: large_conv_model sync interval 2022-03-17T20:57:34.9916898Z 1 with iteration mapping 2022-03-17T20:57:34.9917655Z {0: 0, 1: 3} 2022-03-17T20:57:34.9946623Z Running test: large_conv_model sync interval 2022-03-17T20:57:34.9947648Z 1 with iteration mapping 2022-03-17T20:57:34.9948439Z {0: 0, 1: 10} 2022-03-17T20:57:34.9975933Z Running test: large_conv_model sync interval 2022-03-17T20:57:34.9976911Z 1 with iteration mapping 2022-03-17T20:57:34.9977713Z {0: 5, 1: 7} 2022-03-17T20:57:35.1839464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:35.1841288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:35.9153068Z Running test: large_conv_model sync interval 2022-03-17T20:57:35.9154065Z 1 with iteration mapping 2022-03-17T20:57:35.9154858Z {0: 5, 1: 8} 2022-03-17T20:57:36.1006812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:36.1008367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:36.8317692Z Running test: large_conv_model sync interval 2022-03-17T20:57:36.8318699Z 1 with iteration mapping 2022-03-17T20:57:36.8319488Z {0: 5, 1: 15} 2022-03-17T20:57:37.0169600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.0171118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7482891Z Running test: small_model sync interval 2022-03-17T20:57:37.7483906Z 1 with iteration mapping 2022-03-17T20:57:37.7484720Z {0: 0, 1: 2} 2022-03-17T20:57:37.7502039Z Running test: small_model sync interval 2022-03-17T20:57:37.7503063Z 1 with iteration mapping 2022-03-17T20:57:37.7503889Z {0: 0, 1: 3} 2022-03-17T20:57:37.7518535Z Running test: small_model sync interval 2022-03-17T20:57:37.7519516Z 1 with iteration mapping 2022-03-17T20:57:37.7520642Z {0: 0, 1: 10} 2022-03-17T20:57:37.7537104Z Running test: small_model sync interval 2022-03-17T20:57:37.7538081Z 1 with iteration mapping 2022-03-17T20:57:37.7538867Z {0: 5, 1: 7} 2022-03-17T20:57:37.7576916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7578445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7645253Z Running test: small_model sync interval 2022-03-17T20:57:37.7646251Z 1 with iteration mapping 2022-03-17T20:57:37.7647033Z {0: 5, 1: 8} 2022-03-17T20:57:37.7683877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7685451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7753972Z Running test: small_model sync interval 2022-03-17T20:57:37.7755001Z 1 with iteration mapping 2022-03-17T20:57:37.7755804Z {0: 5, 1: 15} 2022-03-17T20:57:37.7793711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7795275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.7868579Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.7869721Z 1 with iteration mapping 2022-03-17T20:57:37.7870496Z {0: 0, 1: 2} 2022-03-17T20:57:37.7886541Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.7887659Z 1 with iteration mapping 2022-03-17T20:57:37.7888414Z {0: 0, 1: 3} 2022-03-17T20:57:37.7902883Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.7903985Z 1 with iteration mapping 2022-03-17T20:57:37.7904745Z {0: 0, 1: 10} 2022-03-17T20:57:37.7923221Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.7924309Z 1 with iteration mapping 2022-03-17T20:57:37.7925129Z {0: 5, 1: 7} 2022-03-17T20:57:37.8019732Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8020827Z 1 with iteration mapping 2022-03-17T20:57:37.8021908Z {0: 5, 1: 8} 2022-03-17T20:57:37.8139548Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8140702Z 1 with iteration mapping 2022-03-17T20:57:37.8141518Z {0: 5, 1: 15} 2022-03-17T20:57:37.8234534Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8235656Z 1 with iteration mapping 2022-03-17T20:57:37.8236444Z {0: 0, 1: 2} 2022-03-17T20:57:37.8264054Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8265146Z 1 with iteration mapping 2022-03-17T20:57:37.8265906Z {0: 0, 1: 3} 2022-03-17T20:57:37.8291265Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8292337Z 1 with iteration mapping 2022-03-17T20:57:37.8293117Z {0: 0, 1: 10} 2022-03-17T20:57:37.8317289Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8318361Z 1 with iteration mapping 2022-03-17T20:57:37.8319134Z {0: 5, 1: 7} 2022-03-17T20:57:37.8414276Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8415422Z 1 with iteration mapping 2022-03-17T20:57:37.8416252Z {0: 5, 1: 8} 2022-03-17T20:57:37.8527042Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:57:37.8528534Z 1 with iteration mapping 2022-03-17T20:57:37.8529168Z {0: 5, 1: 15} 2022-03-17T20:57:37.8637126Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:37.8638181Z 1 with iteration mapping 2022-03-17T20:57:37.8638976Z {0: 0, 1: 2} 2022-03-17T20:57:37.8661350Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:37.8662407Z 1 with iteration mapping 2022-03-17T20:57:37.8663194Z {0: 0, 1: 3} 2022-03-17T20:57:37.8687273Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:37.8688292Z 1 with iteration mapping 2022-03-17T20:57:37.8689095Z {0: 0, 1: 10} 2022-03-17T20:57:37.8709541Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:37.8710622Z 1 with iteration mapping 2022-03-17T20:57:37.8711418Z {0: 5, 1: 7} 2022-03-17T20:57:37.8761132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.8762883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.8837627Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:37.8838662Z 1 with iteration mapping 2022-03-17T20:57:37.8839460Z {0: 5, 1: 8} 2022-03-17T20:57:37.8887961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.8889555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.8965178Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:57:37.8966210Z 1 with iteration mapping 2022-03-17T20:57:37.8967016Z {0: 5, 1: 15} 2022-03-17T20:57:37.9005387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9006985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9078215Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:37.9079244Z 1 with iteration mapping 2022-03-17T20:57:37.9080034Z {0: 0, 1: 2} 2022-03-17T20:57:37.9098348Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:37.9099400Z 1 with iteration mapping 2022-03-17T20:57:37.9100062Z {0: 0, 1: 3} 2022-03-17T20:57:37.9125433Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:37.9126520Z 1 with iteration mapping 2022-03-17T20:57:37.9127358Z {0: 0, 1: 10} 2022-03-17T20:57:37.9148014Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:37.9149075Z 1 with iteration mapping 2022-03-17T20:57:37.9149832Z {0: 5, 1: 7} 2022-03-17T20:57:37.9195952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9197514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9271335Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:37.9272395Z 1 with iteration mapping 2022-03-17T20:57:37.9273222Z {0: 5, 1: 8} 2022-03-17T20:57:37.9322264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9323832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9402183Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:57:37.9403304Z 1 with iteration mapping 2022-03-17T20:57:37.9404153Z {0: 5, 1: 15} 2022-03-17T20:57:37.9441427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:37.9443161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:40.7733990Z ok (44.686s) 2022-03-17T20:57:40.7734598Z 2022-03-17T20:57:40.7735801Z ---------------------------------------------------------------------- 2022-03-17T20:57:40.7736872Z Ran 1 test in 44.686s 2022-03-17T20:57:40.7737366Z 2022-03-17T20:57:40.7737643Z OK 2022-03-17T20:57:40.7741527Z 2022-03-17T20:57:40.7742109Z Generating XML reports... 2022-03-17T20:57:40.7816075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205656.xml 2022-03-17T20:57:43.4043572Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:57:43.4055626Z 2022-03-17T20:57:43.4055983Z Running tests... 2022-03-17T20:57:43.4056657Z ---------------------------------------------------------------------- 2022-03-17T20:57:44.0061836Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10794 2022-03-17T20:57:44.0088651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10795 2022-03-17T20:57:44.0090274Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:57:45.0780398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:45.1521025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:45.1523581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:45.1589870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:45.1591540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:45.2533445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:45.6367359Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:57:45.6898166Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:57:54.9296867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:54.9304276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:54.9579551Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:54.9581657Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:54.9592464Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:54.9594494Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:55.0316203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:55.0317732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:57:55.0373626Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:57:55.0376127Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:57:57.6415290Z ok (14.235s) 2022-03-17T20:57:57.6415895Z 2022-03-17T20:57:57.6417114Z ---------------------------------------------------------------------- 2022-03-17T20:57:57.6418033Z Ran 1 test in 14.236s 2022-03-17T20:57:57.6418421Z 2022-03-17T20:57:57.6418615Z OK 2022-03-17T20:57:57.6418896Z 2022-03-17T20:57:57.6419188Z Generating XML reports... 2022-03-17T20:57:57.6502120Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205743.xml 2022-03-17T20:58:00.2100519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:58:00.2109344Z 2022-03-17T20:58:00.2109906Z Running tests... 2022-03-17T20:58:00.2111358Z ---------------------------------------------------------------------- 2022-03-17T20:58:00.8081812Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11013 2022-03-17T20:58:00.8116500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11014 2022-03-17T20:58:00.8118149Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:58:01.8744502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:01.8858535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:01.8860515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:01.8947495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:01.8948816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:01.9874332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:02.3731285Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:58:02.4244876Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:58:14.1435450Z ok (13.932s) 2022-03-17T20:58:14.1435910Z 2022-03-17T20:58:14.1437297Z ---------------------------------------------------------------------- 2022-03-17T20:58:14.1438097Z Ran 1 test in 13.932s 2022-03-17T20:58:14.1438462Z 2022-03-17T20:58:14.1438647Z OK 2022-03-17T20:58:14.1438942Z 2022-03-17T20:58:14.1439226Z Generating XML reports... 2022-03-17T20:58:14.1508505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205800.xml 2022-03-17T20:58:16.7087536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:58:16.7098302Z 2022-03-17T20:58:16.7099082Z Running tests... 2022-03-17T20:58:16.7100065Z ---------------------------------------------------------------------- 2022-03-17T20:58:17.3082391Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11232 2022-03-17T20:58:17.3118917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11233 2022-03-17T20:58:17.3120255Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:58:18.4213239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:18.4255137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:18.4257577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:18.4315909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:18.4317572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:18.4319167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:58:18.5266610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:18.5269595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:58:18.5271503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:18.5335196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:20.4203489Z ok (3.710s) 2022-03-17T20:58:20.4203896Z 2022-03-17T20:58:20.4206804Z ---------------------------------------------------------------------- 2022-03-17T20:58:20.4207636Z Ran 1 test in 3.710s 2022-03-17T20:58:20.4208012Z 2022-03-17T20:58:20.4208220Z OK 2022-03-17T20:58:20.4208517Z 2022-03-17T20:58:20.4208821Z Generating XML reports... 2022-03-17T20:58:20.4290719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205816.xml 2022-03-17T20:58:23.0755394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:58:23.0767834Z 2022-03-17T20:58:23.0768293Z Running tests... 2022-03-17T20:58:23.0769456Z ---------------------------------------------------------------------- 2022-03-17T20:58:23.7047333Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11444 2022-03-17T20:58:23.7086401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11445 2022-03-17T20:58:23.7087749Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:58:24.8278588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:24.8279079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:24.8279939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:24.8280752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:24.8283480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:24.8284090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:24.8285626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:58:24.8286690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:58:24.8288082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:24.8390941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:26.7171607Z ok (3.640s) 2022-03-17T20:58:26.7172079Z 2022-03-17T20:58:26.7173121Z ---------------------------------------------------------------------- 2022-03-17T20:58:26.7173879Z Ran 1 test in 3.640s 2022-03-17T20:58:26.7174246Z 2022-03-17T20:58:26.7174432Z OK 2022-03-17T20:58:26.7175513Z 2022-03-17T20:58:26.7183324Z Generating XML reports... 2022-03-17T20:58:26.7257976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205823.xml 2022-03-17T20:58:29.3659484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:58:29.3670035Z 2022-03-17T20:58:29.3670320Z Running tests... 2022-03-17T20:58:29.3672229Z ---------------------------------------------------------------------- 2022-03-17T20:58:29.9668345Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11656 2022-03-17T20:58:29.9699849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11657 2022-03-17T20:58:29.9701182Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:58:31.0816919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:31.1457813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:31.1459724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:31.1530710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:31.1547575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:31.2467031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:31.6264998Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:58:31.6853507Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:58:40.8132583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:58:40.8133902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:58:40.8239722Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:58:40.8244475Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:58:43.3940860Z ok (14.026s) 2022-03-17T20:58:43.3941316Z 2022-03-17T20:58:43.3942315Z ---------------------------------------------------------------------- 2022-03-17T20:58:43.3943075Z Ran 1 test in 14.027s 2022-03-17T20:58:43.3943471Z 2022-03-17T20:58:43.3943666Z OK 2022-03-17T20:58:43.3943962Z 2022-03-17T20:58:43.3944222Z Generating XML reports... 2022-03-17T20:58:43.4021369Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205829.xml 2022-03-17T20:58:46.0135987Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:58:46.0146309Z 2022-03-17T20:58:46.0146876Z Running tests... 2022-03-17T20:58:46.0148782Z ---------------------------------------------------------------------- 2022-03-17T20:58:46.6146198Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11875 2022-03-17T20:58:46.6175745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11876 2022-03-17T20:58:46.6177298Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:58:47.7640873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:47.7771792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:47.7774151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:47.7844903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:47.7846482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:47.8780309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:48.1644355Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:58:48.2618001Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:58:57.3748685Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:58:57.5610558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:58:57.5612051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:00.3444275Z ok (14.329s) 2022-03-17T20:59:00.3444879Z 2022-03-17T20:59:00.3446221Z ---------------------------------------------------------------------- 2022-03-17T20:59:00.3447381Z Ran 1 test in 14.330s 2022-03-17T20:59:00.3448343Z 2022-03-17T20:59:00.3448636Z OK 2022-03-17T20:59:00.3449088Z 2022-03-17T20:59:00.3449462Z Generating XML reports... 2022-03-17T20:59:00.3527737Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205846.xml 2022-03-17T20:59:02.8927121Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:02.8938010Z 2022-03-17T20:59:02.8938286Z Running tests... 2022-03-17T20:59:02.8938754Z ---------------------------------------------------------------------- 2022-03-17T20:59:03.4946825Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12094 2022-03-17T20:59:03.4976445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12095 2022-03-17T20:59:03.4976956Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:04.5621914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:04.6410179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:04.6410949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:04.6429932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:04.6431891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:04.7425399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:06.4072186Z ok (3.513s) 2022-03-17T20:59:06.4072587Z 2022-03-17T20:59:06.4074301Z ---------------------------------------------------------------------- 2022-03-17T20:59:06.4075428Z Ran 1 test in 3.513s 2022-03-17T20:59:06.4075832Z 2022-03-17T20:59:06.4076024Z OK 2022-03-17T20:59:06.4076305Z 2022-03-17T20:59:06.4076586Z Generating XML reports... 2022-03-17T20:59:06.4150331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205902.xml 2022-03-17T20:59:09.0302628Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:09.0315623Z 2022-03-17T20:59:09.0316214Z Running tests... 2022-03-17T20:59:09.0317659Z ---------------------------------------------------------------------- 2022-03-17T20:59:09.0336973Z test_gather (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:59:09.0337702Z 2022-03-17T20:59:09.0338679Z ---------------------------------------------------------------------- 2022-03-17T20:59:09.0339633Z Ran 1 test in 0.002s 2022-03-17T20:59:09.0340084Z 2022-03-17T20:59:09.0340300Z OK (skipped=1) 2022-03-17T20:59:09.0340647Z 2022-03-17T20:59:09.0340934Z Generating XML reports... 2022-03-17T20:59:09.0383635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205909.xml 2022-03-17T20:59:11.7148726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:11.7162025Z 2022-03-17T20:59:11.7162730Z Running tests... 2022-03-17T20:59:11.7164158Z ---------------------------------------------------------------------- 2022-03-17T20:59:11.7185152Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:59:11.7185962Z 2022-03-17T20:59:11.7186924Z ---------------------------------------------------------------------- 2022-03-17T20:59:11.7187875Z Ran 1 test in 0.002s 2022-03-17T20:59:11.7188248Z 2022-03-17T20:59:11.7188483Z OK (skipped=1) 2022-03-17T20:59:11.7188829Z 2022-03-17T20:59:11.7189100Z Generating XML reports... 2022-03-17T20:59:11.7227592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205911.xml 2022-03-17T20:59:14.2757237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:14.2770347Z 2022-03-17T20:59:14.2770800Z Running tests... 2022-03-17T20:59:14.2772200Z ---------------------------------------------------------------------- 2022-03-17T20:59:14.8761700Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12436 2022-03-17T20:59:14.8795517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12437 2022-03-17T20:59:14.8797052Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:15.9986190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:16.0299867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:16.0300922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:16.0394306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:16.0400386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:16.1312835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:16.5197373Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:16.5822764Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:18.3890643Z ok (4.112s) 2022-03-17T20:59:18.3891216Z 2022-03-17T20:59:18.3892565Z ---------------------------------------------------------------------- 2022-03-17T20:59:18.3893616Z Ran 1 test in 4.112s 2022-03-17T20:59:18.3894048Z 2022-03-17T20:59:18.3894326Z OK 2022-03-17T20:59:18.3894689Z 2022-03-17T20:59:18.3895058Z Generating XML reports... 2022-03-17T20:59:18.3973269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205914.xml 2022-03-17T20:59:20.9363194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:20.9377139Z 2022-03-17T20:59:20.9377670Z Running tests... 2022-03-17T20:59:20.9378839Z ---------------------------------------------------------------------- 2022-03-17T20:59:20.9390334Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:59:20.9391124Z 2022-03-17T20:59:20.9392122Z ---------------------------------------------------------------------- 2022-03-17T20:59:20.9393079Z Ran 1 test in 0.002s 2022-03-17T20:59:20.9393455Z 2022-03-17T20:59:20.9393696Z OK (skipped=1) 2022-03-17T20:59:20.9394050Z 2022-03-17T20:59:20.9394307Z Generating XML reports... 2022-03-17T20:59:20.9431157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205920.xml 2022-03-17T20:59:23.4482195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:23.4493784Z 2022-03-17T20:59:23.4494680Z Running tests... 2022-03-17T20:59:23.4496194Z ---------------------------------------------------------------------- 2022-03-17T20:59:23.4506672Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T20:59:23.4507514Z 2022-03-17T20:59:23.4508475Z ---------------------------------------------------------------------- 2022-03-17T20:59:23.4509459Z Ran 1 test in 0.002s 2022-03-17T20:59:23.4509946Z 2022-03-17T20:59:23.4510213Z OK (skipped=1) 2022-03-17T20:59:23.4510553Z 2022-03-17T20:59:23.4510831Z Generating XML reports... 2022-03-17T20:59:23.4548981Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205923.xml 2022-03-17T20:59:26.2394115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:26.2406773Z 2022-03-17T20:59:26.2407493Z Running tests... 2022-03-17T20:59:26.2408070Z ---------------------------------------------------------------------- 2022-03-17T20:59:26.8390151Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12791 2022-03-17T20:59:26.8428015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12792 2022-03-17T20:59:26.8429855Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:27.9569704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:27.9575292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:27.9577272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:27.9673014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:27.9674364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:28.0592101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:28.7675854Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:28.8227751Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:30.5539963Z ok (4.313s) 2022-03-17T20:59:30.5540661Z 2022-03-17T20:59:30.5542013Z ---------------------------------------------------------------------- 2022-03-17T20:59:30.5542725Z Ran 1 test in 4.313s 2022-03-17T20:59:30.5543057Z 2022-03-17T20:59:30.5543232Z OK 2022-03-17T20:59:30.5543559Z 2022-03-17T20:59:30.5543856Z Generating XML reports... 2022-03-17T20:59:30.5626505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205926.xml 2022-03-17T20:59:33.1822059Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:33.1834440Z 2022-03-17T20:59:33.1834775Z Running tests... 2022-03-17T20:59:33.1835309Z ---------------------------------------------------------------------- 2022-03-17T20:59:33.7821056Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13006 2022-03-17T20:59:33.7850046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13007 2022-03-17T20:59:33.7851435Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:34.9047449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:34.9048109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:34.9049014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:34.9049817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:34.9053669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:34.9054226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:34.9266675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:59:34.9267840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:59:34.9270199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:59:34.9373914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:59:35.6642950Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:35.7197358Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:35.9647631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:59:35.9650113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:59:35.9651708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:59:35.9653497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:59:36.0098797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:59:36.0101040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:59:36.0102424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:59:36.0104105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:59:37.6952576Z ok (4.511s) 2022-03-17T20:59:37.6953070Z 2022-03-17T20:59:37.6954510Z ---------------------------------------------------------------------- 2022-03-17T20:59:37.6955341Z Ran 1 test in 4.512s 2022-03-17T20:59:37.6955690Z 2022-03-17T20:59:37.6955880Z OK 2022-03-17T20:59:37.6956171Z 2022-03-17T20:59:37.6956452Z Generating XML reports... 2022-03-17T20:59:37.7021500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205933.xml 2022-03-17T20:59:40.3802584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:40.3815599Z 2022-03-17T20:59:40.3816287Z Running tests... 2022-03-17T20:59:40.3817722Z ---------------------------------------------------------------------- 2022-03-17T20:59:40.9812660Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13260 2022-03-17T20:59:40.9843387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13261 2022-03-17T20:59:40.9844957Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:42.0729619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:42.0806965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:42.0809172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:42.0833128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:42.0836584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:42.0838899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:59:42.1814304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:42.1815838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:59:42.1818009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:59:42.1856129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:59:43.8923743Z ok (3.510s) 2022-03-17T20:59:43.8924333Z 2022-03-17T20:59:43.8925447Z ---------------------------------------------------------------------- 2022-03-17T20:59:43.8926417Z Ran 1 test in 3.511s 2022-03-17T20:59:43.8926862Z 2022-03-17T20:59:43.8927074Z OK 2022-03-17T20:59:43.8927458Z 2022-03-17T20:59:43.8927797Z Generating XML reports... 2022-03-17T20:59:43.9011008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205940.xml 2022-03-17T20:59:46.4489275Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:46.4501424Z 2022-03-17T20:59:46.4501606Z Running tests... 2022-03-17T20:59:46.4502222Z ---------------------------------------------------------------------- 2022-03-17T20:59:47.0479748Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13472 2022-03-17T20:59:47.0515424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13473 2022-03-17T20:59:47.0516672Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:48.1175119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:48.1804572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:48.1806517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:48.1890164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:48.1897069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:48.2815100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:48.5718287Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:48.6590782Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T20:59:51.0625237Z ok (4.612s) 2022-03-17T20:59:51.0625840Z 2022-03-17T20:59:51.0627019Z ---------------------------------------------------------------------- 2022-03-17T20:59:51.0628028Z Ran 1 test in 4.612s 2022-03-17T20:59:51.0628458Z 2022-03-17T20:59:51.0628693Z OK 2022-03-17T20:59:51.0629082Z 2022-03-17T20:59:51.0629384Z Generating XML reports... 2022-03-17T20:59:51.0705777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205946.xml 2022-03-17T20:59:53.6130497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:53.6142814Z 2022-03-17T20:59:53.6143098Z Running tests... 2022-03-17T20:59:53.6143772Z ---------------------------------------------------------------------- 2022-03-17T20:59:54.2184205Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13687 2022-03-17T20:59:54.2226008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13688 2022-03-17T20:59:54.2227277Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T20:59:55.3494444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:55.3496784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:55.3498150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:55.3500325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:55.3501665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:55.3502872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:57.1327952Z ok (3.518s) 2022-03-17T20:59:57.1328643Z 2022-03-17T20:59:57.1329735Z ---------------------------------------------------------------------- 2022-03-17T20:59:57.1330544Z Ran 1 test in 3.518s 2022-03-17T20:59:57.1330931Z 2022-03-17T20:59:57.1331146Z OK 2022-03-17T20:59:57.1331449Z 2022-03-17T20:59:57.1331738Z Generating XML reports... 2022-03-17T20:59:57.1414421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205953.xml 2022-03-17T20:59:59.7701847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:59:59.7714673Z 2022-03-17T20:59:59.7715329Z Running tests... 2022-03-17T20:59:59.7716795Z ---------------------------------------------------------------------- 2022-03-17T21:00:00.3937064Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13895 2022-03-17T21:00:00.3984781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13896 2022-03-17T21:00:00.3986369Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:00:01.4619391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:01.4818034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:01.4819213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:01.4820061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:01.4822751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:01.4824965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:00:01.5829476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:01.5831696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:00:01.5833878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:00:01.5839844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:00:03.3069270Z ok (3.535s) 2022-03-17T21:00:03.3069642Z 2022-03-17T21:00:03.3071801Z ---------------------------------------------------------------------- 2022-03-17T21:00:03.3072828Z Ran 1 test in 3.536s 2022-03-17T21:00:03.3073220Z 2022-03-17T21:00:03.3073409Z OK 2022-03-17T21:00:03.3073687Z 2022-03-17T21:00:03.3073982Z Generating XML reports... 2022-03-17T21:00:03.3144733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205959.xml 2022-03-17T21:00:05.9032252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:05.9045625Z 2022-03-17T21:00:05.9046052Z Running tests... 2022-03-17T21:00:05.9047140Z ---------------------------------------------------------------------- 2022-03-17T21:00:06.5398921Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14107 2022-03-17T21:00:06.5445544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14108 2022-03-17T21:00:06.5446924Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:00:07.6704882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:07.6976261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:07.6978161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:07.7010602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:07.7021270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:07.7023567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:00:07.7986439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:07.7987677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:00:07.7989480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:00:07.8050397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:00:09.6571077Z ok (3.752s) 2022-03-17T21:00:09.6571768Z 2022-03-17T21:00:09.6572824Z ---------------------------------------------------------------------- 2022-03-17T21:00:09.6573614Z Ran 1 test in 3.752s 2022-03-17T21:00:09.6574010Z 2022-03-17T21:00:09.6574179Z OK 2022-03-17T21:00:09.6574477Z 2022-03-17T21:00:09.6574756Z Generating XML reports... 2022-03-17T21:00:09.6635012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210005.xml 2022-03-17T21:00:12.2762991Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:12.2778822Z 2022-03-17T21:00:12.2779092Z Running tests... 2022-03-17T21:00:12.2779643Z ---------------------------------------------------------------------- 2022-03-17T21:00:12.8782523Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14319 2022-03-17T21:00:12.8818261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14320 2022-03-17T21:00:12.8819612Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:00:13.9670656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:14.0268292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:14.0270408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:14.0281750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:14.0292676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:14.1281323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:14.4093982Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:00:14.5099458Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:00:26.5117107Z ok (14.233s) 2022-03-17T21:00:26.5117848Z 2022-03-17T21:00:26.5119387Z ---------------------------------------------------------------------- 2022-03-17T21:00:26.5120337Z Ran 1 test in 14.233s 2022-03-17T21:00:26.5120790Z 2022-03-17T21:00:26.5122505Z OK 2022-03-17T21:00:26.5123415Z 2022-03-17T21:00:26.5124558Z Generating XML reports... 2022-03-17T21:00:26.5192363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210012.xml 2022-03-17T21:00:29.1776028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:29.1790084Z 2022-03-17T21:00:29.1790859Z Running tests... 2022-03-17T21:00:29.1792290Z ---------------------------------------------------------------------- 2022-03-17T21:00:29.1815309Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:00:29.1816075Z 2022-03-17T21:00:29.1816858Z ---------------------------------------------------------------------- 2022-03-17T21:00:29.1817663Z Ran 1 test in 0.003s 2022-03-17T21:00:29.1818040Z 2022-03-17T21:00:29.1818305Z OK (skipped=1) 2022-03-17T21:00:29.1818788Z 2022-03-17T21:00:29.1819187Z Generating XML reports... 2022-03-17T21:00:29.1859335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210029.xml 2022-03-17T21:00:31.9039227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:31.9050995Z 2022-03-17T21:00:31.9051793Z Running tests... 2022-03-17T21:00:31.9052980Z ---------------------------------------------------------------------- 2022-03-17T21:00:31.9065891Z test_isend (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:00:31.9066866Z 2022-03-17T21:00:31.9068100Z ---------------------------------------------------------------------- 2022-03-17T21:00:31.9068911Z Ran 1 test in 0.002s 2022-03-17T21:00:31.9069296Z 2022-03-17T21:00:31.9069558Z OK (skipped=1) 2022-03-17T21:00:31.9069922Z 2022-03-17T21:00:31.9070199Z Generating XML reports... 2022-03-17T21:00:31.9107221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210031.xml 2022-03-17T21:00:34.5772073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:34.5784472Z 2022-03-17T21:00:34.5785003Z Running tests... 2022-03-17T21:00:34.5786176Z ---------------------------------------------------------------------- 2022-03-17T21:00:34.5801603Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:00:34.5802625Z 2022-03-17T21:00:34.5803911Z ---------------------------------------------------------------------- 2022-03-17T21:00:34.5804730Z Ran 1 test in 0.002s 2022-03-17T21:00:34.5805084Z 2022-03-17T21:00:34.5805321Z OK (skipped=1) 2022-03-17T21:00:34.5805675Z 2022-03-17T21:00:34.5805945Z Generating XML reports... 2022-03-17T21:00:34.5847192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210034.xml 2022-03-17T21:00:37.2021738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:37.2038703Z 2022-03-17T21:00:37.2038999Z Running tests... 2022-03-17T21:00:37.2039445Z ---------------------------------------------------------------------- 2022-03-17T21:00:37.2055442Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:00:37.2055885Z 2022-03-17T21:00:37.2056547Z ---------------------------------------------------------------------- 2022-03-17T21:00:37.2056964Z Ran 1 test in 0.002s 2022-03-17T21:00:37.2057174Z 2022-03-17T21:00:37.2057343Z OK (skipped=1) 2022-03-17T21:00:37.2057565Z 2022-03-17T21:00:37.2057734Z Generating XML reports... 2022-03-17T21:00:37.2095308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210037.xml 2022-03-17T21:00:39.9867751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:39.9882531Z 2022-03-17T21:00:39.9883112Z Running tests... 2022-03-17T21:00:39.9885095Z ---------------------------------------------------------------------- 2022-03-17T21:00:40.6007762Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14806 2022-03-17T21:00:40.6053543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14807 2022-03-17T21:00:40.6055172Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:00:41.7260645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:41.7309576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:41.7311899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:41.7361275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:41.7365064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:41.8326980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:43.5134142Z skip (3.525s) 2022-03-17T21:00:43.5134562Z 2022-03-17T21:00:43.5136638Z ---------------------------------------------------------------------- 2022-03-17T21:00:43.5137611Z Ran 1 test in 3.525s 2022-03-17T21:00:43.5138621Z 2022-03-17T21:00:43.5138887Z OK (skipped=1) 2022-03-17T21:00:43.5139243Z 2022-03-17T21:00:43.5139501Z Generating XML reports... 2022-03-17T21:00:43.5223692Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210039.xml 2022-03-17T21:00:46.3310294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:46.3324467Z 2022-03-17T21:00:46.3324966Z Running tests... 2022-03-17T21:00:46.3326175Z ---------------------------------------------------------------------- 2022-03-17T21:00:46.9380117Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15014 2022-03-17T21:00:46.9420924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15015 2022-03-17T21:00:46.9422977Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:00:48.0164219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:48.0701582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:48.0703670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:48.0772878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:48.0776850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:48.1713266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:49.6522765Z skip (3.320s) 2022-03-17T21:00:49.6523442Z 2022-03-17T21:00:49.6525365Z ---------------------------------------------------------------------- 2022-03-17T21:00:49.6526198Z Ran 1 test in 3.320s 2022-03-17T21:00:49.6526569Z 2022-03-17T21:00:49.6526808Z OK (skipped=1) 2022-03-17T21:00:49.6527177Z 2022-03-17T21:00:49.6527429Z Generating XML reports... 2022-03-17T21:00:49.6609098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210046.xml 2022-03-17T21:00:52.2949864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:52.2965741Z 2022-03-17T21:00:52.2966809Z Running tests... 2022-03-17T21:00:52.2967964Z ---------------------------------------------------------------------- 2022-03-17T21:00:52.2986301Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:00:52.2987247Z 2022-03-17T21:00:52.2988416Z ---------------------------------------------------------------------- 2022-03-17T21:00:52.2989473Z Ran 1 test in 0.002s 2022-03-17T21:00:52.2989868Z 2022-03-17T21:00:52.2990108Z OK (skipped=1) 2022-03-17T21:00:52.2990482Z 2022-03-17T21:00:52.2990740Z Generating XML reports... 2022-03-17T21:00:52.3028904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210052.xml 2022-03-17T21:00:54.9743834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:54.9758105Z 2022-03-17T21:00:54.9758384Z Running tests... 2022-03-17T21:00:54.9759020Z ---------------------------------------------------------------------- 2022-03-17T21:00:54.9786278Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:00:54.9786615Z 2022-03-17T21:00:54.9786978Z ---------------------------------------------------------------------- 2022-03-17T21:00:54.9787331Z Ran 1 test in 0.003s 2022-03-17T21:00:54.9787504Z 2022-03-17T21:00:54.9787624Z OK (skipped=1) 2022-03-17T21:00:54.9787772Z 2022-03-17T21:00:54.9787878Z Generating XML reports... 2022-03-17T21:00:54.9831626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210054.xml 2022-03-17T21:00:57.7043865Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:00:57.7059367Z 2022-03-17T21:00:57.7060104Z Running tests... 2022-03-17T21:00:57.7061472Z ---------------------------------------------------------------------- 2022-03-17T21:00:57.7082970Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:00:57.7084030Z 2022-03-17T21:00:57.7085012Z ---------------------------------------------------------------------- 2022-03-17T21:00:57.7085957Z Ran 1 test in 0.002s 2022-03-17T21:00:57.7086415Z 2022-03-17T21:00:57.7086729Z OK (skipped=1) 2022-03-17T21:00:57.7087108Z 2022-03-17T21:00:57.7087372Z Generating XML reports... 2022-03-17T21:00:57.7121810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210057.xml 2022-03-17T21:01:00.4153358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:00.4170190Z 2022-03-17T21:01:00.4170341Z Running tests... 2022-03-17T21:01:00.4170879Z ---------------------------------------------------------------------- 2022-03-17T21:01:00.4193655Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:01:00.4193939Z 2022-03-17T21:01:00.4194301Z ---------------------------------------------------------------------- 2022-03-17T21:01:00.4194601Z Ran 1 test in 0.002s 2022-03-17T21:01:00.4194737Z 2022-03-17T21:01:00.4194830Z OK (skipped=1) 2022-03-17T21:01:00.4195005Z 2022-03-17T21:01:00.4195134Z Generating XML reports... 2022-03-17T21:01:00.4234871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210100.xml 2022-03-17T21:01:03.1162187Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:03.1178701Z 2022-03-17T21:01:03.1179355Z Running tests... 2022-03-17T21:01:03.1180535Z ---------------------------------------------------------------------- 2022-03-17T21:01:03.1196280Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:01:03.1197221Z 2022-03-17T21:01:03.1198127Z ---------------------------------------------------------------------- 2022-03-17T21:01:03.1198938Z Ran 1 test in 0.002s 2022-03-17T21:01:03.1199667Z 2022-03-17T21:01:03.1199929Z OK (skipped=1) 2022-03-17T21:01:03.1200286Z 2022-03-17T21:01:03.1200562Z Generating XML reports... 2022-03-17T21:01:03.1236895Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210103.xml 2022-03-17T21:01:05.8248433Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:05.8261153Z 2022-03-17T21:01:05.8261588Z Running tests... 2022-03-17T21:01:05.8263025Z ---------------------------------------------------------------------- 2022-03-17T21:01:06.4312008Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15557 2022-03-17T21:01:06.4352055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15558 2022-03-17T21:01:06.4353401Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:07.5740725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:07.5742037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:07.5743996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:07.5745881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:07.5747612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:07.5748773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:07.9593692Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:08.0109312Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:09.7455544Z ok (3.919s) 2022-03-17T21:01:09.7456078Z 2022-03-17T21:01:09.7457110Z ---------------------------------------------------------------------- 2022-03-17T21:01:09.7458052Z Ran 1 test in 3.919s 2022-03-17T21:01:09.7458494Z 2022-03-17T21:01:09.7458725Z OK 2022-03-17T21:01:09.7459005Z 2022-03-17T21:01:09.7459279Z Generating XML reports... 2022-03-17T21:01:09.7541483Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210105.xml 2022-03-17T21:01:12.5540928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:12.5558579Z 2022-03-17T21:01:12.5558810Z Running tests... 2022-03-17T21:01:12.5559244Z ---------------------------------------------------------------------- 2022-03-17T21:01:13.1616860Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15772 2022-03-17T21:01:13.1657583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15773 2022-03-17T21:01:13.1658985Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:14.2281391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:14.3038896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:14.3039763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:14.3089453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:14.3094524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:14.4052830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:14.6854292Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:14.7848890Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:16.7761813Z ok (4.220s) 2022-03-17T21:01:16.7762903Z 2022-03-17T21:01:16.7764154Z ---------------------------------------------------------------------- 2022-03-17T21:01:16.7765084Z Ran 1 test in 4.220s 2022-03-17T21:01:16.7765533Z 2022-03-17T21:01:16.7765795Z OK 2022-03-17T21:01:16.7766145Z 2022-03-17T21:01:16.7766464Z Generating XML reports... 2022-03-17T21:01:16.7847661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210112.xml 2022-03-17T21:01:19.3724796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:19.3738091Z 2022-03-17T21:01:19.3738390Z Running tests... 2022-03-17T21:01:19.3738824Z ---------------------------------------------------------------------- 2022-03-17T21:01:19.9779784Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15987 2022-03-17T21:01:19.9813141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15988 2022-03-17T21:01:19.9814476Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:21.0677750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:21.0854314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:21.0856161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:21.0880824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:21.0882751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:21.1862810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:21.5651430Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:21.6163343Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:23.2911380Z ok (3.917s) 2022-03-17T21:01:23.2912121Z 2022-03-17T21:01:23.2914331Z ---------------------------------------------------------------------- 2022-03-17T21:01:23.2915307Z Ran 1 test in 3.917s 2022-03-17T21:01:23.2915686Z 2022-03-17T21:01:23.2915877Z OK 2022-03-17T21:01:23.2916172Z 2022-03-17T21:01:23.2916466Z Generating XML reports... 2022-03-17T21:01:23.2980726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210119.xml 2022-03-17T21:01:25.9290910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:25.9305054Z 2022-03-17T21:01:25.9305929Z Running tests... 2022-03-17T21:01:25.9307365Z ---------------------------------------------------------------------- 2022-03-17T21:01:26.5468553Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16202 2022-03-17T21:01:26.5541460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16203 2022-03-17T21:01:26.5546520Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:27.6693393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:27.6738717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:27.6741175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:27.6794057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:27.6802518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:27.7751502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:28.0623245Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:28.1575897Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:29.9648994Z ok (4.034s) 2022-03-17T21:01:29.9649377Z 2022-03-17T21:01:29.9650439Z ---------------------------------------------------------------------- 2022-03-17T21:01:29.9651210Z Ran 1 test in 4.034s 2022-03-17T21:01:29.9651579Z 2022-03-17T21:01:29.9651767Z OK 2022-03-17T21:01:29.9652054Z 2022-03-17T21:01:29.9652354Z Generating XML reports... 2022-03-17T21:01:29.9739940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210125.xml 2022-03-17T21:01:32.6272561Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:32.6289501Z 2022-03-17T21:01:32.6289882Z Running tests... 2022-03-17T21:01:32.6291199Z ---------------------------------------------------------------------- 2022-03-17T21:01:33.2329525Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16417 2022-03-17T21:01:33.2365870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16418 2022-03-17T21:01:33.2367218Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:34.3797918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:34.3798628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:34.3799704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:34.3897614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:34.3899309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:34.4812359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:34.8621110Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:34.9172449Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:01:36.8464016Z ok (4.217s) 2022-03-17T21:01:36.8464424Z 2022-03-17T21:01:36.8466706Z ---------------------------------------------------------------------- 2022-03-17T21:01:36.8467573Z Ran 1 test in 4.218s 2022-03-17T21:01:36.8467950Z 2022-03-17T21:01:36.8468152Z OK 2022-03-17T21:01:36.8468454Z 2022-03-17T21:01:36.8468724Z Generating XML reports... 2022-03-17T21:01:36.8548791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210132.xml 2022-03-17T21:01:39.4703771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:39.4715623Z 2022-03-17T21:01:39.4715879Z Running tests... 2022-03-17T21:01:39.4717012Z ---------------------------------------------------------------------- 2022-03-17T21:01:39.4740708Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:01:39.4741349Z 2022-03-17T21:01:39.4742131Z ---------------------------------------------------------------------- 2022-03-17T21:01:39.4742904Z Ran 1 test in 0.002s 2022-03-17T21:01:39.4743722Z 2022-03-17T21:01:39.4743993Z OK (skipped=1) 2022-03-17T21:01:39.4744332Z 2022-03-17T21:01:39.4744604Z Generating XML reports... 2022-03-17T21:01:39.4781242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210139.xml 2022-03-17T21:01:42.7928775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:42.7943220Z 2022-03-17T21:01:42.7943515Z Running tests... 2022-03-17T21:01:42.7944749Z ---------------------------------------------------------------------- 2022-03-17T21:01:42.7970627Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:01:42.7971314Z 2022-03-17T21:01:42.7972115Z ---------------------------------------------------------------------- 2022-03-17T21:01:42.7972889Z Ran 1 test in 0.003s 2022-03-17T21:01:42.7973259Z 2022-03-17T21:01:42.7973469Z OK (skipped=1) 2022-03-17T21:01:42.7973819Z 2022-03-17T21:01:42.7974107Z Generating XML reports... 2022-03-17T21:01:42.8014493Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210142.xml 2022-03-17T21:01:45.4481825Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:45.4493579Z 2022-03-17T21:01:45.4493927Z Running tests... 2022-03-17T21:01:45.4495105Z ---------------------------------------------------------------------- 2022-03-17T21:01:45.4514454Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:01:45.4515235Z 2022-03-17T21:01:45.4516070Z ---------------------------------------------------------------------- 2022-03-17T21:01:45.4516844Z Ran 1 test in 0.002s 2022-03-17T21:01:45.4517221Z 2022-03-17T21:01:45.4517451Z OK (skipped=1) 2022-03-17T21:01:45.4517809Z 2022-03-17T21:01:45.4518089Z Generating XML reports... 2022-03-17T21:01:45.4553646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210145.xml 2022-03-17T21:01:48.1692938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:48.1707925Z 2022-03-17T21:01:48.1708369Z Running tests... 2022-03-17T21:01:48.1709662Z ---------------------------------------------------------------------- 2022-03-17T21:01:48.7974841Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16846 2022-03-17T21:01:48.8011018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16847 2022-03-17T21:01:49.9435939Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:49.9437274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:49.9543513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:49.9545425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:49.9640335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:49.9641709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:50.0554649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:51.7106247Z ok (3.539s) 2022-03-17T21:01:51.7106724Z 2022-03-17T21:01:51.7107718Z ---------------------------------------------------------------------- 2022-03-17T21:01:51.7108496Z Ran 1 test in 3.540s 2022-03-17T21:01:51.7108839Z 2022-03-17T21:01:51.7109033Z OK 2022-03-17T21:01:51.7109326Z 2022-03-17T21:01:51.7109613Z Generating XML reports... 2022-03-17T21:01:51.7194612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210148.xml 2022-03-17T21:01:54.3259674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:01:54.3275363Z 2022-03-17T21:01:54.3276382Z Running tests... 2022-03-17T21:01:54.3277546Z ---------------------------------------------------------------------- 2022-03-17T21:01:54.9327995Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17054 2022-03-17T21:01:54.9364557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17055 2022-03-17T21:01:54.9365897Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:01:56.0131165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:56.0406765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:56.0408718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:56.0435517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:56.0436884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:56.1418471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:57.8461074Z ok (3.518s) 2022-03-17T21:01:57.8461474Z 2022-03-17T21:01:57.8462353Z ---------------------------------------------------------------------- 2022-03-17T21:01:57.8463104Z Ran 1 test in 3.519s 2022-03-17T21:01:57.8463469Z 2022-03-17T21:01:57.8463676Z OK 2022-03-17T21:01:57.8463973Z 2022-03-17T21:01:57.8464248Z Generating XML reports... 2022-03-17T21:01:57.8544950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210154.xml 2022-03-17T21:02:00.4250796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:02:00.4263417Z 2022-03-17T21:02:00.4263793Z Running tests... 2022-03-17T21:02:00.4264467Z ---------------------------------------------------------------------- 2022-03-17T21:02:00.4287149Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:02:00.4287455Z 2022-03-17T21:02:00.4287825Z ---------------------------------------------------------------------- 2022-03-17T21:02:00.4288184Z Ran 1 test in 0.002s 2022-03-17T21:02:00.4288353Z 2022-03-17T21:02:00.4288446Z OK (skipped=1) 2022-03-17T21:02:00.4288608Z 2022-03-17T21:02:00.4288747Z Generating XML reports... 2022-03-17T21:02:00.4326432Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210200.xml 2022-03-17T21:02:03.0256005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:02:03.0269643Z 2022-03-17T21:02:03.0270357Z Running tests... 2022-03-17T21:02:03.0271574Z ---------------------------------------------------------------------- 2022-03-17T21:02:03.0288120Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:02:03.0288878Z 2022-03-17T21:02:03.0289695Z ---------------------------------------------------------------------- 2022-03-17T21:02:03.0290464Z Ran 1 test in 0.002s 2022-03-17T21:02:03.0290812Z 2022-03-17T21:02:03.0291046Z OK (skipped=1) 2022-03-17T21:02:03.0291396Z 2022-03-17T21:02:03.0291665Z Generating XML reports... 2022-03-17T21:02:03.0328079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210203.xml 2022-03-17T21:02:05.7066663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:02:05.7081429Z 2022-03-17T21:02:05.7081762Z Running tests... 2022-03-17T21:02:05.7083300Z ---------------------------------------------------------------------- 2022-03-17T21:02:06.3361768Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17396 2022-03-17T21:02:06.3402481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17397 2022-03-17T21:02:06.3403807Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:02:07.4080114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:07.4153039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:07.4154967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:07.4182636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:07.4183986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:07.5166062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:07.8972776Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:07.9503713Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:19.7689390Z ok (14.060s) 2022-03-17T21:02:19.7690259Z 2022-03-17T21:02:19.7692509Z ---------------------------------------------------------------------- 2022-03-17T21:02:19.7693407Z Ran 1 test in 14.060s 2022-03-17T21:02:19.7693809Z 2022-03-17T21:02:19.7693988Z OK 2022-03-17T21:02:19.7694415Z 2022-03-17T21:02:19.7694729Z Generating XML reports... 2022-03-17T21:02:19.7766733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210205.xml 2022-03-17T21:02:22.4546138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:02:22.4560395Z 2022-03-17T21:02:22.4560976Z Running tests... 2022-03-17T21:02:22.4563197Z ---------------------------------------------------------------------- 2022-03-17T21:02:23.0609852Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17615 2022-03-17T21:02:23.0644103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17616 2022-03-17T21:02:23.0645556Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:02:24.1856390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:24.2001334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:24.2003733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:24.2060474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:24.2066314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:24.3013055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:24.5926298Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:24.6858439Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:36.7901733Z ok (14.334s) 2022-03-17T21:02:36.7903223Z 2022-03-17T21:02:36.7904743Z ---------------------------------------------------------------------- 2022-03-17T21:02:36.7905678Z Ran 1 test in 14.334s 2022-03-17T21:02:36.7906081Z 2022-03-17T21:02:36.7908105Z OK 2022-03-17T21:02:36.7908346Z 2022-03-17T21:02:36.7908920Z Generating XML reports... 2022-03-17T21:02:36.7985248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210222.xml 2022-03-17T21:02:39.4380632Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:02:39.4396150Z 2022-03-17T21:02:39.4396795Z Running tests... 2022-03-17T21:02:39.4397971Z ---------------------------------------------------------------------- 2022-03-17T21:02:40.0483430Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17834 2022-03-17T21:02:40.0521634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17835 2022-03-17T21:02:40.0523118Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:02:41.1450348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:41.1581937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:41.1583926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:41.1656460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:41.1661066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:41.2589259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:41.7877729Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:42.0558152Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:43.8631741Z ok (4.423s) 2022-03-17T21:02:43.8632175Z 2022-03-17T21:02:43.8636384Z ---------------------------------------------------------------------- 2022-03-17T21:02:43.8637347Z Ran 1 test in 4.424s 2022-03-17T21:02:43.8637699Z 2022-03-17T21:02:43.8637922Z OK 2022-03-17T21:02:43.8638241Z 2022-03-17T21:02:43.8638603Z Generating XML reports... 2022-03-17T21:02:43.8705872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210239.xml 2022-03-17T21:02:46.5271487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:02:46.5286991Z 2022-03-17T21:02:46.5287304Z Running tests... 2022-03-17T21:02:46.5287752Z ---------------------------------------------------------------------- 2022-03-17T21:02:47.1356189Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18049 2022-03-17T21:02:47.1394180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18050 2022-03-17T21:02:47.1395637Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:02:48.2586049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:48.2737052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:48.2739481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:48.2787878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:48.2790001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:48.3751296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:48.6565482Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:48.7620462Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:02:58.2192455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:58.2470071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:58.2488736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:58.2493475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:00.8733475Z ok (14.344s) 2022-03-17T21:03:00.8734106Z 2022-03-17T21:03:00.8735562Z ---------------------------------------------------------------------- 2022-03-17T21:03:00.8736694Z Ran 1 test in 14.345s 2022-03-17T21:03:00.8737254Z 2022-03-17T21:03:00.8739378Z OK 2022-03-17T21:03:00.8739801Z 2022-03-17T21:03:00.8740193Z Generating XML reports... 2022-03-17T21:03:00.8819293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210246.xml 2022-03-17T21:03:03.4979653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:03.4989706Z 2022-03-17T21:03:03.4990344Z Running tests... 2022-03-17T21:03:03.4991916Z ---------------------------------------------------------------------- 2022-03-17T21:03:03.5009531Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:03.5010400Z 2022-03-17T21:03:03.5011346Z ---------------------------------------------------------------------- 2022-03-17T21:03:03.5012367Z Ran 1 test in 0.002s 2022-03-17T21:03:03.5012858Z 2022-03-17T21:03:03.5013140Z OK (skipped=1) 2022-03-17T21:03:03.5013476Z 2022-03-17T21:03:03.5013770Z Generating XML reports... 2022-03-17T21:03:03.5050289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210303.xml 2022-03-17T21:03:06.2209699Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:06.2220950Z 2022-03-17T21:03:06.2221420Z Running tests... 2022-03-17T21:03:06.2222735Z ---------------------------------------------------------------------- 2022-03-17T21:03:06.2241120Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:06.2242784Z 2022-03-17T21:03:06.2243736Z ---------------------------------------------------------------------- 2022-03-17T21:03:06.2244772Z Ran 1 test in 0.002s 2022-03-17T21:03:06.2245270Z 2022-03-17T21:03:06.2245549Z OK (skipped=1) 2022-03-17T21:03:06.2245883Z 2022-03-17T21:03:06.2246181Z Generating XML reports... 2022-03-17T21:03:06.2286682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210306.xml 2022-03-17T21:03:08.9569820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:08.9581428Z 2022-03-17T21:03:08.9581762Z Running tests... 2022-03-17T21:03:08.9582225Z ---------------------------------------------------------------------- 2022-03-17T21:03:08.9602769Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:08.9603047Z 2022-03-17T21:03:08.9605438Z ---------------------------------------------------------------------- 2022-03-17T21:03:08.9606512Z Ran 1 test in 0.002s 2022-03-17T21:03:08.9606990Z 2022-03-17T21:03:08.9607284Z OK (skipped=1) 2022-03-17T21:03:08.9607700Z 2022-03-17T21:03:08.9608000Z Generating XML reports... 2022-03-17T21:03:08.9645039Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210308.xml 2022-03-17T21:03:11.4786596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:11.4800398Z 2022-03-17T21:03:11.4800946Z Running tests... 2022-03-17T21:03:11.4801534Z ---------------------------------------------------------------------- 2022-03-17T21:03:11.4821626Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:11.4821936Z 2022-03-17T21:03:11.4822394Z ---------------------------------------------------------------------- 2022-03-17T21:03:11.4822852Z Ran 1 test in 0.002s 2022-03-17T21:03:11.4823075Z 2022-03-17T21:03:11.4823237Z OK (skipped=1) 2022-03-17T21:03:11.4826784Z 2022-03-17T21:03:11.4827005Z Generating XML reports... 2022-03-17T21:03:11.4859162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210311.xml 2022-03-17T21:03:14.0619246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:14.0630496Z 2022-03-17T21:03:14.0631090Z Running tests... 2022-03-17T21:03:14.0632361Z ---------------------------------------------------------------------- 2022-03-17T21:03:14.0649053Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:14.0649806Z 2022-03-17T21:03:14.0650824Z ---------------------------------------------------------------------- 2022-03-17T21:03:14.0651800Z Ran 1 test in 0.002s 2022-03-17T21:03:14.0652755Z 2022-03-17T21:03:14.0653031Z OK (skipped=1) 2022-03-17T21:03:14.0653416Z 2022-03-17T21:03:14.0653722Z Generating XML reports... 2022-03-17T21:03:14.0688768Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210314.xml 2022-03-17T21:03:16.7426717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:16.7439254Z 2022-03-17T21:03:16.7439573Z Running tests... 2022-03-17T21:03:16.7440050Z ---------------------------------------------------------------------- 2022-03-17T21:03:16.7453692Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:16.7453953Z 2022-03-17T21:03:16.7454484Z ---------------------------------------------------------------------- 2022-03-17T21:03:16.7454781Z Ran 1 test in 0.002s 2022-03-17T21:03:16.7454932Z 2022-03-17T21:03:16.7455044Z OK (skipped=1) 2022-03-17T21:03:16.7455356Z 2022-03-17T21:03:16.7455739Z Generating XML reports... 2022-03-17T21:03:16.7496058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210316.xml 2022-03-17T21:03:19.8105883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:19.8119344Z 2022-03-17T21:03:19.8120072Z Running tests... 2022-03-17T21:03:19.8121301Z ---------------------------------------------------------------------- 2022-03-17T21:03:19.8140979Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:19.8141710Z 2022-03-17T21:03:19.8142559Z ---------------------------------------------------------------------- 2022-03-17T21:03:19.8143366Z Ran 1 test in 0.002s 2022-03-17T21:03:19.8143783Z 2022-03-17T21:03:19.8144047Z OK (skipped=1) 2022-03-17T21:03:19.8144434Z 2022-03-17T21:03:19.8144696Z Generating XML reports... 2022-03-17T21:03:19.8186006Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210319.xml 2022-03-17T21:03:22.5776661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:22.5785971Z 2022-03-17T21:03:22.5786358Z Running tests... 2022-03-17T21:03:22.5786918Z ---------------------------------------------------------------------- 2022-03-17T21:03:22.5806214Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:22.5806535Z 2022-03-17T21:03:22.5807180Z ---------------------------------------------------------------------- 2022-03-17T21:03:22.5807641Z Ran 1 test in 0.002s 2022-03-17T21:03:22.5807796Z 2022-03-17T21:03:22.5807917Z OK (skipped=1) 2022-03-17T21:03:22.5808090Z 2022-03-17T21:03:22.5808221Z Generating XML reports... 2022-03-17T21:03:22.5846788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210322.xml 2022-03-17T21:03:25.3155999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:25.3169311Z 2022-03-17T21:03:25.3169755Z Running tests... 2022-03-17T21:03:25.3170434Z ---------------------------------------------------------------------- 2022-03-17T21:03:25.3189576Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:25.3190108Z 2022-03-17T21:03:25.3190619Z ---------------------------------------------------------------------- 2022-03-17T21:03:25.3191178Z Ran 1 test in 0.002s 2022-03-17T21:03:25.3191429Z 2022-03-17T21:03:25.3191610Z OK (skipped=1) 2022-03-17T21:03:25.3191911Z 2022-03-17T21:03:25.3192023Z Generating XML reports... 2022-03-17T21:03:25.3234367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210325.xml 2022-03-17T21:03:28.0448596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:28.0459340Z 2022-03-17T21:03:28.0459669Z Running tests... 2022-03-17T21:03:28.0481408Z ---------------------------------------------------------------------- 2022-03-17T21:03:28.0482221Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:28.0482561Z 2022-03-17T21:03:28.0483008Z ---------------------------------------------------------------------- 2022-03-17T21:03:28.0483453Z Ran 1 test in 0.002s 2022-03-17T21:03:28.0483661Z 2022-03-17T21:03:28.0483778Z OK (skipped=1) 2022-03-17T21:03:28.0483952Z 2022-03-17T21:03:28.0484090Z Generating XML reports... 2022-03-17T21:03:28.0524709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210328.xml 2022-03-17T21:03:30.7224645Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:30.7237341Z 2022-03-17T21:03:30.7238103Z Running tests... 2022-03-17T21:03:30.7239555Z ---------------------------------------------------------------------- 2022-03-17T21:03:31.3516094Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18938 2022-03-17T21:03:31.3548535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18939 2022-03-17T21:03:31.3550106Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:03:32.4992701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:32.5071814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:32.5073956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:32.5096752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:32.5102961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:32.6089526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:32.8940797Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:03:32.9984514Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:03:34.8634420Z ok (4.139s) 2022-03-17T21:03:34.8635399Z 2022-03-17T21:03:34.8637231Z ---------------------------------------------------------------------- 2022-03-17T21:03:34.8638062Z Ran 1 test in 4.139s 2022-03-17T21:03:34.8638473Z 2022-03-17T21:03:34.8638698Z OK 2022-03-17T21:03:34.8639018Z 2022-03-17T21:03:34.8639329Z Generating XML reports... 2022-03-17T21:03:34.8706963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210330.xml 2022-03-17T21:03:37.5534719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:37.5546340Z 2022-03-17T21:03:37.5546596Z Running tests... 2022-03-17T21:03:37.5547063Z ---------------------------------------------------------------------- 2022-03-17T21:03:37.5564561Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:37.5564881Z 2022-03-17T21:03:37.5565291Z ---------------------------------------------------------------------- 2022-03-17T21:03:37.5565668Z Ran 1 test in 0.002s 2022-03-17T21:03:37.5565858Z 2022-03-17T21:03:37.5566026Z OK (skipped=1) 2022-03-17T21:03:37.5566227Z 2022-03-17T21:03:37.5566380Z Generating XML reports... 2022-03-17T21:03:37.5604541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210337.xml 2022-03-17T21:03:40.1652762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:40.1667002Z 2022-03-17T21:03:40.1667519Z Running tests... 2022-03-17T21:03:40.1668717Z ---------------------------------------------------------------------- 2022-03-17T21:03:40.1686905Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:40.1687629Z 2022-03-17T21:03:40.1688604Z ---------------------------------------------------------------------- 2022-03-17T21:03:40.1689674Z Ran 1 test in 0.002s 2022-03-17T21:03:40.1690171Z 2022-03-17T21:03:40.1690496Z OK (skipped=1) 2022-03-17T21:03:40.1690928Z 2022-03-17T21:03:40.1691243Z Generating XML reports... 2022-03-17T21:03:40.1733741Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210340.xml 2022-03-17T21:03:42.8434860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:42.8447696Z 2022-03-17T21:03:42.8448005Z Running tests... 2022-03-17T21:03:42.8448491Z ---------------------------------------------------------------------- 2022-03-17T21:03:43.4433551Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19293 2022-03-17T21:03:43.4463707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19294 2022-03-17T21:03:43.4464847Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:03:44.5251967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:44.5637991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:44.5639946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:44.5662615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:44.5671664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:44.6647238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:44.9536997Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:03:45.0454912Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:03:46.8566137Z ok (4.011s) 2022-03-17T21:03:46.8566712Z 2022-03-17T21:03:46.8567990Z ---------------------------------------------------------------------- 2022-03-17T21:03:46.8569837Z Ran 1 test in 4.012s 2022-03-17T21:03:46.8570470Z 2022-03-17T21:03:46.8570652Z OK 2022-03-17T21:03:46.8570983Z 2022-03-17T21:03:46.8571298Z Generating XML reports... 2022-03-17T21:03:46.8653840Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210342.xml 2022-03-17T21:03:49.4898094Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:49.4910610Z 2022-03-17T21:03:49.4911092Z Running tests... 2022-03-17T21:03:49.4911535Z ---------------------------------------------------------------------- 2022-03-17T21:03:50.0898017Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19514 2022-03-17T21:03:50.0927117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19515 2022-03-17T21:03:50.0927846Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:03:51.1955750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:51.2555608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:51.2556414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:51.2562405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:51.2568045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:51.3566660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:51.7470933Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:03:51.7977317Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:03:54.0028729Z ok (4.511s) 2022-03-17T21:03:54.0029244Z 2022-03-17T21:03:54.0030244Z ---------------------------------------------------------------------- 2022-03-17T21:03:54.0031108Z Ran 1 test in 4.512s 2022-03-17T21:03:54.0031475Z 2022-03-17T21:03:54.0031705Z OK 2022-03-17T21:03:54.0032052Z 2022-03-17T21:03:54.0032368Z Generating XML reports... 2022-03-17T21:03:54.0106777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210349.xml 2022-03-17T21:03:56.6680478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:56.6691643Z 2022-03-17T21:03:56.6691917Z Running tests... 2022-03-17T21:03:56.6692400Z ---------------------------------------------------------------------- 2022-03-17T21:03:56.6710351Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:56.6710623Z 2022-03-17T21:03:56.6710947Z ---------------------------------------------------------------------- 2022-03-17T21:03:56.6711314Z Ran 1 test in 0.002s 2022-03-17T21:03:56.6711581Z 2022-03-17T21:03:56.6711734Z OK (skipped=1) 2022-03-17T21:03:56.6711908Z 2022-03-17T21:03:56.6712071Z Generating XML reports... 2022-03-17T21:03:56.6751238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210356.xml 2022-03-17T21:03:59.2152852Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:03:59.2167913Z 2022-03-17T21:03:59.2168508Z Running tests... 2022-03-17T21:03:59.2169717Z ---------------------------------------------------------------------- 2022-03-17T21:03:59.2183512Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:03:59.2184335Z 2022-03-17T21:03:59.2185374Z ---------------------------------------------------------------------- 2022-03-17T21:03:59.2186368Z Ran 1 test in 0.002s 2022-03-17T21:03:59.2187327Z 2022-03-17T21:03:59.2187621Z OK (skipped=1) 2022-03-17T21:03:59.2188021Z 2022-03-17T21:03:59.2188330Z Generating XML reports... 2022-03-17T21:03:59.2226763Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210359.xml 2022-03-17T21:04:01.8889542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:01.8900489Z 2022-03-17T21:04:01.8900942Z Running tests... 2022-03-17T21:04:01.8902127Z ---------------------------------------------------------------------- 2022-03-17T21:04:01.8925783Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:04:01.8926342Z 2022-03-17T21:04:01.8927023Z ---------------------------------------------------------------------- 2022-03-17T21:04:01.8927710Z Ran 1 test in 0.002s 2022-03-17T21:04:01.8928098Z 2022-03-17T21:04:01.8928349Z OK (skipped=1) 2022-03-17T21:04:01.8928723Z 2022-03-17T21:04:01.8928991Z Generating XML reports... 2022-03-17T21:04:01.8965623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210401.xml 2022-03-17T21:04:04.5004614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:04.5019244Z 2022-03-17T21:04:04.5019473Z Running tests... 2022-03-17T21:04:04.5019953Z ---------------------------------------------------------------------- 2022-03-17T21:04:04.5034794Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:04:04.5035303Z 2022-03-17T21:04:04.5035695Z ---------------------------------------------------------------------- 2022-03-17T21:04:04.5036105Z Ran 1 test in 0.002s 2022-03-17T21:04:04.5036329Z 2022-03-17T21:04:04.5036426Z OK (skipped=1) 2022-03-17T21:04:04.5036602Z 2022-03-17T21:04:04.5036742Z Generating XML reports... 2022-03-17T21:04:04.5075279Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210404.xml 2022-03-17T21:04:07.1026730Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:07.1040388Z 2022-03-17T21:04:07.1040750Z Running tests... 2022-03-17T21:04:07.1041228Z ---------------------------------------------------------------------- 2022-03-17T21:04:07.7140465Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20003 2022-03-17T21:04:07.7174607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20004 2022-03-17T21:04:07.7176055Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:04:08.8000054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:08.8232076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:08.8234510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:08.8306273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:08.8314466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:08.9241125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:09.2181231Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:09.3007053Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:11.1267318Z ok (4.022s) 2022-03-17T21:04:11.1268019Z 2022-03-17T21:04:11.1269462Z ---------------------------------------------------------------------- 2022-03-17T21:04:11.1270527Z Ran 1 test in 4.022s 2022-03-17T21:04:11.1271017Z 2022-03-17T21:04:11.1272045Z OK 2022-03-17T21:04:11.1272435Z 2022-03-17T21:04:11.1272880Z Generating XML reports... 2022-03-17T21:04:11.1341861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210407.xml 2022-03-17T21:04:13.8223894Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:13.8237036Z 2022-03-17T21:04:13.8237363Z Running tests... 2022-03-17T21:04:14.4437999Z ---------------------------------------------------------------------- 2022-03-17T21:04:14.4439118Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20224 2022-03-17T21:04:14.4475406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20225 2022-03-17T21:04:14.4476455Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:04:15.5589540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:15.5800180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:15.5829416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:15.5894501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:15.5900127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:15.6813014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:15.9752522Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:16.0653800Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:18.2577810Z ok (4.434s) 2022-03-17T21:04:18.2578257Z 2022-03-17T21:04:18.2579401Z ---------------------------------------------------------------------- 2022-03-17T21:04:18.2580469Z Ran 1 test in 4.434s 2022-03-17T21:04:18.2580965Z 2022-03-17T21:04:18.2581238Z OK 2022-03-17T21:04:18.2581616Z 2022-03-17T21:04:18.2581941Z Generating XML reports... 2022-03-17T21:04:18.2664649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210413.xml 2022-03-17T21:04:20.8981968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:20.8994686Z 2022-03-17T21:04:20.8994841Z Running tests... 2022-03-17T21:04:20.8995424Z ---------------------------------------------------------------------- 2022-03-17T21:04:20.9011678Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:04:20.9012027Z 2022-03-17T21:04:20.9012566Z ---------------------------------------------------------------------- 2022-03-17T21:04:20.9013011Z Ran 1 test in 0.002s 2022-03-17T21:04:20.9013167Z 2022-03-17T21:04:20.9013294Z OK (skipped=1) 2022-03-17T21:04:20.9013466Z 2022-03-17T21:04:20.9013612Z Generating XML reports... 2022-03-17T21:04:20.9047981Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210420.xml 2022-03-17T21:04:23.3964599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:23.3973810Z 2022-03-17T21:04:23.3974734Z Running tests... 2022-03-17T21:04:23.3976127Z ---------------------------------------------------------------------- 2022-03-17T21:04:23.3988942Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:04:23.4041076Z 2022-03-17T21:04:23.4042079Z ---------------------------------------------------------------------- 2022-03-17T21:04:23.4042963Z Ran 1 test in 0.002s 2022-03-17T21:04:23.4043314Z 2022-03-17T21:04:23.4043584Z OK (skipped=1) 2022-03-17T21:04:23.4044438Z 2022-03-17T21:04:23.4044752Z Generating XML reports... 2022-03-17T21:04:23.4046375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210423.xml 2022-03-17T21:04:25.9768728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:25.9780700Z 2022-03-17T21:04:25.9781077Z Running tests... 2022-03-17T21:04:25.9781519Z ---------------------------------------------------------------------- 2022-03-17T21:04:25.9804017Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:04:25.9804276Z 2022-03-17T21:04:25.9804734Z ---------------------------------------------------------------------- 2022-03-17T21:04:25.9805104Z Ran 1 test in 0.002s 2022-03-17T21:04:25.9805306Z 2022-03-17T21:04:25.9805429Z OK (skipped=1) 2022-03-17T21:04:25.9805614Z 2022-03-17T21:04:25.9805759Z Generating XML reports... 2022-03-17T21:04:25.9848537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210425.xml 2022-03-17T21:04:28.7126300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:28.7137972Z 2022-03-17T21:04:28.7138797Z Running tests... 2022-03-17T21:04:28.7140060Z ---------------------------------------------------------------------- 2022-03-17T21:04:28.7152848Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:04:28.7154444Z 2022-03-17T21:04:28.7155291Z ---------------------------------------------------------------------- 2022-03-17T21:04:28.7156092Z Ran 1 test in 0.002s 2022-03-17T21:04:28.7156495Z 2022-03-17T21:04:28.7156755Z OK (skipped=1) 2022-03-17T21:04:28.7157130Z 2022-03-17T21:04:28.7157429Z Generating XML reports... 2022-03-17T21:04:28.7194424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210428.xml 2022-03-17T21:04:31.1949532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:31.1961285Z 2022-03-17T21:04:31.1961590Z Running tests... 2022-03-17T21:04:31.1962132Z ---------------------------------------------------------------------- 2022-03-17T21:04:31.1978084Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:04:31.1979014Z 2022-03-17T21:04:31.1979811Z ---------------------------------------------------------------------- 2022-03-17T21:04:31.1980642Z Ran 1 test in 0.002s 2022-03-17T21:04:31.1981051Z 2022-03-17T21:04:31.1981302Z OK (skipped=1) 2022-03-17T21:04:31.1981681Z 2022-03-17T21:04:31.1981978Z Generating XML reports... 2022-03-17T21:04:31.2018788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210431.xml 2022-03-17T21:04:33.7985896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:33.7999989Z 2022-03-17T21:04:33.8000535Z Running tests... 2022-03-17T21:04:33.8001037Z ---------------------------------------------------------------------- 2022-03-17T21:04:33.8018284Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:04:33.8018726Z 2022-03-17T21:04:33.8019292Z ---------------------------------------------------------------------- 2022-03-17T21:04:33.8019709Z Ran 1 test in 0.002s 2022-03-17T21:04:33.8019890Z 2022-03-17T21:04:33.8019993Z OK (skipped=1) 2022-03-17T21:04:33.8020662Z 2022-03-17T21:04:33.8072032Z Generating XML reports... 2022-03-17T21:04:33.8073591Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210433.xml 2022-03-17T21:04:36.5152899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:36.5167100Z 2022-03-17T21:04:36.5167410Z Running tests... 2022-03-17T21:04:36.5168069Z ---------------------------------------------------------------------- 2022-03-17T21:04:36.5187305Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:04:36.5187586Z 2022-03-17T21:04:36.5189121Z ---------------------------------------------------------------------- 2022-03-17T21:04:36.5189933Z Ran 1 test in 0.002s 2022-03-17T21:04:36.5190281Z 2022-03-17T21:04:36.5190510Z OK (skipped=1) 2022-03-17T21:04:36.5190798Z 2022-03-17T21:04:36.5191068Z Generating XML reports... 2022-03-17T21:04:36.5227408Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210436.xml 2022-03-17T21:04:39.0298550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:39.0313232Z 2022-03-17T21:04:39.0313601Z Running tests... 2022-03-17T21:04:39.0314853Z ---------------------------------------------------------------------- 2022-03-17T21:04:39.0328581Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:04:39.0329311Z 2022-03-17T21:04:39.0330168Z ---------------------------------------------------------------------- 2022-03-17T21:04:39.0331091Z Ran 1 test in 0.002s 2022-03-17T21:04:39.0331487Z 2022-03-17T21:04:39.0331704Z OK (skipped=1) 2022-03-17T21:04:39.0332065Z 2022-03-17T21:04:39.0332371Z Generating XML reports... 2022-03-17T21:04:39.0371326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210439.xml 2022-03-17T21:04:41.5026991Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:41.5042489Z 2022-03-17T21:04:41.5042707Z Running tests... 2022-03-17T21:04:41.5043406Z ---------------------------------------------------------------------- 2022-03-17T21:04:42.1015746Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20981 2022-03-17T21:04:42.1059316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20982 2022-03-17T21:04:42.1060686Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:04:43.2820365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:43.2934004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:43.3021719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:43.3022581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:43.3032545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:43.3949794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:43.6835386Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:43.7808410Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:45.6206295Z ok (4.110s) 2022-03-17T21:04:45.6206798Z 2022-03-17T21:04:45.6207801Z ---------------------------------------------------------------------- 2022-03-17T21:04:45.6208601Z Ran 1 test in 4.111s 2022-03-17T21:04:45.6209011Z 2022-03-17T21:04:45.6209269Z OK 2022-03-17T21:04:45.6209593Z 2022-03-17T21:04:45.6209903Z Generating XML reports... 2022-03-17T21:04:45.6228282Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210441.xml 2022-03-17T21:04:48.2571780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:48.2588291Z 2022-03-17T21:04:48.2588563Z Running tests... 2022-03-17T21:04:48.2589247Z ---------------------------------------------------------------------- 2022-03-17T21:04:48.8792772Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21196 2022-03-17T21:04:48.8828046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21197 2022-03-17T21:04:48.8829280Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:04:50.0129222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:50.0134156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:50.0134906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:50.0231630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:50.0232686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:50.1151324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:50.4956915Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:50.5521390Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:52.3935084Z ok (4.134s) 2022-03-17T21:04:52.3935510Z 2022-03-17T21:04:52.3936956Z ---------------------------------------------------------------------- 2022-03-17T21:04:52.3937849Z Ran 1 test in 4.135s 2022-03-17T21:04:52.3938261Z 2022-03-17T21:04:52.3938427Z OK 2022-03-17T21:04:52.3938745Z 2022-03-17T21:04:52.3939050Z Generating XML reports... 2022-03-17T21:04:52.4025977Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210448.xml 2022-03-17T21:04:55.0136283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:04:55.0149966Z 2022-03-17T21:04:55.0150353Z Running tests... 2022-03-17T21:04:55.0151631Z ---------------------------------------------------------------------- 2022-03-17T21:04:55.6362714Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21417 2022-03-17T21:04:55.6402008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21418 2022-03-17T21:04:55.6403393Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:04:56.7265614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:56.7329643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:56.7331754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:56.7370022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:56.7371718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:56.8342133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:57.1177436Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:57.2167248Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:04:58.9486510Z ok (3.933s) 2022-03-17T21:04:58.9487328Z 2022-03-17T21:04:58.9488819Z ---------------------------------------------------------------------- 2022-03-17T21:04:58.9489850Z Ran 1 test in 3.933s 2022-03-17T21:04:58.9490423Z 2022-03-17T21:04:58.9490727Z OK 2022-03-17T21:04:58.9491269Z 2022-03-17T21:04:58.9492495Z Generating XML reports... 2022-03-17T21:04:58.9576120Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210455.xml 2022-03-17T21:05:01.5264370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:01.5277945Z 2022-03-17T21:05:01.5278430Z Running tests... 2022-03-17T21:05:01.5279483Z ---------------------------------------------------------------------- 2022-03-17T21:05:01.5296026Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:05:01.5296809Z 2022-03-17T21:05:01.5297501Z ---------------------------------------------------------------------- 2022-03-17T21:05:01.5298158Z Ran 1 test in 0.002s 2022-03-17T21:05:01.5298523Z 2022-03-17T21:05:01.5298772Z OK (skipped=1) 2022-03-17T21:05:01.5299143Z 2022-03-17T21:05:01.5299396Z Generating XML reports... 2022-03-17T21:05:01.5338712Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210501.xml 2022-03-17T21:05:04.2032360Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:04.2046136Z 2022-03-17T21:05:04.2046436Z Running tests... 2022-03-17T21:05:04.2046877Z ---------------------------------------------------------------------- 2022-03-17T21:05:04.2064459Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:05:04.2064953Z 2022-03-17T21:05:04.2065393Z ---------------------------------------------------------------------- 2022-03-17T21:05:04.2065727Z Ran 1 test in 0.002s 2022-03-17T21:05:04.2065910Z 2022-03-17T21:05:04.2066025Z OK (skipped=1) 2022-03-17T21:05:04.2066207Z 2022-03-17T21:05:04.2066347Z Generating XML reports... 2022-03-17T21:05:04.2109474Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210504.xml 2022-03-17T21:05:06.8995226Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:06.9009580Z 2022-03-17T21:05:06.9010334Z Running tests... 2022-03-17T21:05:06.9011613Z ---------------------------------------------------------------------- 2022-03-17T21:05:06.9025530Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:05:06.9077477Z 2022-03-17T21:05:06.9078198Z ---------------------------------------------------------------------- 2022-03-17T21:05:06.9078947Z Ran 1 test in 0.002s 2022-03-17T21:05:06.9079249Z 2022-03-17T21:05:06.9079477Z OK (skipped=1) 2022-03-17T21:05:06.9079794Z 2022-03-17T21:05:06.9080049Z Generating XML reports... 2022-03-17T21:05:06.9081395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210506.xml 2022-03-17T21:05:09.3962443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:09.3973902Z 2022-03-17T21:05:09.3974431Z Running tests... 2022-03-17T21:05:09.3975516Z ---------------------------------------------------------------------- 2022-03-17T21:05:09.3992961Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:05:09.3993894Z 2022-03-17T21:05:09.3994819Z ---------------------------------------------------------------------- 2022-03-17T21:05:09.3995851Z Ran 1 test in 0.002s 2022-03-17T21:05:09.3996338Z 2022-03-17T21:05:09.3996585Z OK (skipped=1) 2022-03-17T21:05:09.3996957Z 2022-03-17T21:05:09.3997283Z Generating XML reports... 2022-03-17T21:05:09.4036060Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210509.xml 2022-03-17T21:05:11.8885318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:11.8897434Z 2022-03-17T21:05:11.8897797Z Running tests... 2022-03-17T21:05:11.8898591Z ---------------------------------------------------------------------- 2022-03-17T21:05:11.8910945Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:05:11.8911282Z 2022-03-17T21:05:11.8911654Z ---------------------------------------------------------------------- 2022-03-17T21:05:11.8911959Z Ran 1 test in 0.002s 2022-03-17T21:05:11.8912117Z 2022-03-17T21:05:11.8912244Z OK (skipped=1) 2022-03-17T21:05:11.8912403Z 2022-03-17T21:05:11.8912518Z Generating XML reports... 2022-03-17T21:05:11.8952683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210511.xml 2022-03-17T21:05:14.4849033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:14.4863197Z 2022-03-17T21:05:14.4863305Z Running tests... 2022-03-17T21:05:14.4863895Z ---------------------------------------------------------------------- 2022-03-17T21:05:14.4881474Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:05:14.4881785Z 2022-03-17T21:05:14.4882394Z ---------------------------------------------------------------------- 2022-03-17T21:05:14.4882771Z Ran 1 test in 0.002s 2022-03-17T21:05:14.4882953Z 2022-03-17T21:05:14.4883054Z OK (skipped=1) 2022-03-17T21:05:14.4883226Z 2022-03-17T21:05:14.4883364Z Generating XML reports... 2022-03-17T21:05:14.4924501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210514.xml 2022-03-17T21:05:17.1467676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:17.1480510Z 2022-03-17T21:05:17.1480822Z Running tests... 2022-03-17T21:05:17.1481270Z ---------------------------------------------------------------------- 2022-03-17T21:05:17.7471830Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22040 2022-03-17T21:05:17.7508630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22041 2022-03-17T21:05:17.7509816Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:05:18.8263364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:05:18.8428148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:05:18.8428987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:05:18.8465114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:05:18.8469157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:05:18.9439545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:05:19.3265553Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:05:19.3788907Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:05:31.1835303Z ok (14.035s) 2022-03-17T21:05:31.1835679Z 2022-03-17T21:05:31.1836594Z ---------------------------------------------------------------------- 2022-03-17T21:05:31.1837448Z Ran 1 test in 14.035s 2022-03-17T21:05:31.1837839Z 2022-03-17T21:05:31.1838026Z OK 2022-03-17T21:05:31.1838350Z 2022-03-17T21:05:31.1842948Z Generating XML reports... 2022-03-17T21:05:31.1926349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210517.xml 2022-03-17T21:05:33.8632530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:33.8645049Z 2022-03-17T21:05:33.8645345Z Running tests... 2022-03-17T21:05:33.8646227Z ---------------------------------------------------------------------- 2022-03-17T21:05:33.8666492Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:05:33.8718527Z 2022-03-17T21:05:33.8719240Z ---------------------------------------------------------------------- 2022-03-17T21:05:33.8719921Z Ran 1 test in 0.002s 2022-03-17T21:05:33.8720252Z 2022-03-17T21:05:33.8720470Z OK (skipped=1) 2022-03-17T21:05:33.8720784Z 2022-03-17T21:05:33.8720999Z Generating XML reports... 2022-03-17T21:05:33.8722490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210533.xml 2022-03-17T21:05:36.4996555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:36.5011472Z 2022-03-17T21:05:36.5011643Z Running tests... 2022-03-17T21:05:36.5013015Z ---------------------------------------------------------------------- 2022-03-17T21:05:37.1280968Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22326 2022-03-17T21:05:37.1323074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22327 2022-03-17T21:05:37.1324708Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:05:38.2330572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:05:38.2513606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:05:38.2515737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:05:38.2532444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:05:38.2543429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:05:38.3523713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:05:38.7351437Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:05:38.7945715Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:05:40.5432118Z ok (4.042s) 2022-03-17T21:05:40.5432574Z 2022-03-17T21:05:40.5434096Z ---------------------------------------------------------------------- 2022-03-17T21:05:40.5435134Z Ran 1 test in 4.042s 2022-03-17T21:05:40.5435526Z 2022-03-17T21:05:40.5435748Z OK 2022-03-17T21:05:40.5436075Z 2022-03-17T21:05:40.5436387Z Generating XML reports... 2022-03-17T21:05:40.5506584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210536.xml 2022-03-17T21:05:43.2373844Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:05:43.2390568Z 2022-03-17T21:05:43.2391190Z Running tests... 2022-03-17T21:05:43.2392572Z ---------------------------------------------------------------------- 2022-03-17T21:05:43.8857679Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22541 2022-03-17T21:05:43.8903080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22542 2022-03-17T21:05:43.8904422Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:05:45.0165203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:05:45.0194919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:05:45.0196582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:05:45.0266039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:05:45.0269520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:05:45.1207739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:05:45.4113031Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:05:45.5004977Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:05:54.6795979Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:05:54.7298669Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:05:57.5138493Z ok (14.274s) 2022-03-17T21:05:57.5140598Z 2022-03-17T21:05:57.5141790Z ---------------------------------------------------------------------- 2022-03-17T21:05:57.5143033Z Ran 1 test in 14.275s 2022-03-17T21:05:57.5143529Z 2022-03-17T21:05:57.5143749Z OK 2022-03-17T21:05:57.5144186Z 2022-03-17T21:05:57.5144670Z Generating XML reports... 2022-03-17T21:05:57.5221921Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210543.xml 2022-03-17T21:06:00.0763071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:06:00.0777851Z 2022-03-17T21:06:00.0778202Z Running tests... 2022-03-17T21:06:00.0778651Z ---------------------------------------------------------------------- 2022-03-17T21:06:00.6761697Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22760 2022-03-17T21:06:00.6802981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22761 2022-03-17T21:06:00.6804387Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:01.7557212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:01.7965802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:01.7966881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:01.7967707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:01.7968339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:01.7969249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:03.4894729Z skip (3.411s) 2022-03-17T21:06:03.4895452Z 2022-03-17T21:06:03.4896801Z ---------------------------------------------------------------------- 2022-03-17T21:06:03.4897714Z Ran 1 test in 3.411s 2022-03-17T21:06:03.4898366Z 2022-03-17T21:06:03.4898659Z OK (skipped=1) 2022-03-17T21:06:03.4899890Z 2022-03-17T21:06:03.4900236Z Generating XML reports... 2022-03-17T21:06:03.4981632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210600.xml 2022-03-17T21:06:06.0338381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T21:06:06.0350674Z 2022-03-17T21:06:06.0351218Z Running tests... 2022-03-17T21:06:06.0351698Z ---------------------------------------------------------------------- 2022-03-17T21:06:06.6345991Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22968 2022-03-17T21:06:06.6375220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22969 2022-03-17T21:06:06.6376667Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:07.7586820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:07.7624283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:07.7625177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:07.7689270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:07.7692561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:07.8639331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:09.6452045Z skip (3.610s) 2022-03-17T21:06:09.6452240Z 2022-03-17T21:06:09.6452775Z ---------------------------------------------------------------------- 2022-03-17T21:06:09.6453513Z Ran 1 test in 3.610s 2022-03-17T21:06:09.6453718Z 2022-03-17T21:06:09.6453897Z OK (skipped=1) 2022-03-17T21:06:09.6454124Z 2022-03-17T21:06:09.6454308Z Generating XML reports... 2022-03-17T21:06:09.6538112Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210606.xml 2022-03-17T21:06:12.6723151Z Running distributed tests for the gloo backend with env init_method 2022-03-17T21:06:12.6725795Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:06:12.671416] 2022-03-17T21:06:13.8117303Z 2022-03-17T21:06:13.8207408Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T21:06:13.8268172Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8268670Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8269081Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8269469Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8269905Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8270338Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8270805Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8271294Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8271783Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8272377Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8272906Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8273419Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8273945Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8274437Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8274904Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8275386Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8275820Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8276242Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8276645Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8277076Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8277548Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8277999Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8278411Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8278783Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8279202Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8279620Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8280009Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8280427Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8280853Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8281287Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8281788Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8282414Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8282956Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8283603Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8284116Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8284649Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8285138Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8285634Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8286121Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8286650Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8287149Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8287653Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8288141Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8288623Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8289124Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8289581Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8290080Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8290584Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8291139Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8291617Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8292087Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8292575Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8293044Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8293501Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8293971Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8294411Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8294857Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8295317Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8295710Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8296075Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8296412Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8296782Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8297174Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8297541Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8297915Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8298259Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8298621Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8298981Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8299356Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8299737Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8300119Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8300486Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8300821Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8301181Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8301551Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8301905Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8302326Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8302685Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8303053Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8303433Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8303820Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8304231Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8304638Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8305061Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8305489Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8305894Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8306313Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8306713Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8307112Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8307526Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8307937Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8308423Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8308836Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8309264Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8309683Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8310077Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8310446Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8310789Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8311139Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8311483Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8311829Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8312207Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8312559Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8312925Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8313304Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8313671Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8314051Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8314409Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8314793Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8315177Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8315571Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8315961Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8316340Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8316730Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8317099Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8317489Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8317860Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8318245Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8318621Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8318976Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8319350Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8319723Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8320131Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8320621Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8321038Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8321424Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8321889Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8322302Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8322719Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8323206Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8323707Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8324197Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8324681Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8325218Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8325617Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8326069Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8326519Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8326963Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8327439Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8327914Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8328418Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8328897Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8329394Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8329979Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8330591Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8331304Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8331993Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8332667Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8333355Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8334072Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8334751Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8335442Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8336042Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8336493Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8336912Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8337257Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8337632Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8338010Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8338366Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8338756Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8339136Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8339553Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8339974Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8340341Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8340713Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8341084Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8341495Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8341950Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8342323Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8342721Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8343104Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8343508Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8343907Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8344283Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8344677Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8345053Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8345432Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8345829Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8346250Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8346651Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8346993Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8347372Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8347764Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8348140Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8348510Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8348841Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8349195Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8349537Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8349902Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8350260Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8350608Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8350968Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8351298Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8351639Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8352032Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8352393Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8352767Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8353099Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8353405Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8353737Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8354097Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8354490Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8354906Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8355323Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8355707Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8356105Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8356517Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8356903Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8357309Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8357680Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8358063Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8358503Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8358872Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8359246Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8359606Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8360046Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8360517Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8360944Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8361367Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8361789Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8362363Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8362772Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8363197Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8363676Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8364126Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8364588Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8365042Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8365490Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8365932Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8366352Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8366797Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8367250Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8367655Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8368068Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8368485Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8368928Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8369352Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8369836Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8370281Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8370692Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8371116Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8371518Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8371974Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8372404Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8372829Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8373286Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8373699Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8374139Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8374564Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8374982Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8375472Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8375901Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8376311Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8376688Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8377102Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8377513Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8377886Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8378260Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8378631Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8379053Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8379447Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8379808Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8380193Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8380547Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8380907Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8381308Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8381714Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:13.8382136Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:06:14.9259086Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:14.9269704Z 2022-03-17T21:06:14.9270499Z Running tests... 2022-03-17T21:06:14.9272078Z ---------------------------------------------------------------------- 2022-03-17T21:06:15.5231590Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23243 2022-03-17T21:06:15.5266134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23244 2022-03-17T21:06:15.5267801Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:16.6619017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:16.6621541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:16.6622933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:16.6624910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:16.6729644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:16.7639458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:17.0700799Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.0704030Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.1652085Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.1654880Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.5622760Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.5625915Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.5627837Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.5630485Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.5730799Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.5732607Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.5735143Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.5737674Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.5836985Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.5839809Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:17.5841600Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:06:17.5844065Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:06:19.3369958Z ok (4.409s) 2022-03-17T21:06:19.3370751Z 2022-03-17T21:06:19.3372141Z ---------------------------------------------------------------------- 2022-03-17T21:06:19.3373198Z Ran 1 test in 4.410s 2022-03-17T21:06:19.3373670Z 2022-03-17T21:06:19.3373940Z OK 2022-03-17T21:06:19.3374338Z 2022-03-17T21:06:19.3374786Z Generating XML reports... 2022-03-17T21:06:19.3451006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210614.xml 2022-03-17T21:06:21.9362748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:21.9374334Z 2022-03-17T21:06:21.9374847Z Running tests... 2022-03-17T21:06:21.9376147Z ---------------------------------------------------------------------- 2022-03-17T21:06:21.9414399Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T21:06:21.9415944Z 2022-03-17T21:06:21.9417079Z ---------------------------------------------------------------------- 2022-03-17T21:06:21.9418110Z Ran 1 test in 0.004s 2022-03-17T21:06:21.9418543Z 2022-03-17T21:06:21.9418854Z OK (skipped=1) 2022-03-17T21:06:21.9419311Z 2022-03-17T21:06:21.9419624Z Generating XML reports... 2022-03-17T21:06:21.9457460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210621.xml 2022-03-17T21:06:24.6571338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:24.6583022Z 2022-03-17T21:06:24.6583178Z Running tests... 2022-03-17T21:06:24.6583859Z ---------------------------------------------------------------------- 2022-03-17T21:06:25.2722777Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23520 2022-03-17T21:06:25.2765570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23521 2022-03-17T21:06:25.2767018Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:26.3802088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:26.3803707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:26.3805651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:26.3807907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:26.3907599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:26.4820285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:28.8860951Z ok (4.227s) 2022-03-17T21:06:28.8861366Z 2022-03-17T21:06:28.8862656Z ---------------------------------------------------------------------- 2022-03-17T21:06:28.8863795Z Ran 1 test in 4.228s 2022-03-17T21:06:28.8864183Z 2022-03-17T21:06:28.8864412Z OK 2022-03-17T21:06:28.8864734Z 2022-03-17T21:06:28.8865026Z Generating XML reports... 2022-03-17T21:06:28.8942973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210624.xml 2022-03-17T21:06:31.6103014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:31.6114322Z 2022-03-17T21:06:31.6114612Z Running tests... 2022-03-17T21:06:31.6116032Z ---------------------------------------------------------------------- 2022-03-17T21:06:32.2276784Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23730 2022-03-17T21:06:32.2314913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23731 2022-03-17T21:06:32.2316468Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:33.3385761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:33.3387917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:33.3389358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:33.3391103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:33.3396304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:33.4400628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:43.2050932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.2052353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.2308712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.2310156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.2395647Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:43.2398217Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:43.2436472Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:43.2438925Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:43.3713489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.3715344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.5215275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.5216749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.5473732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.5475053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.6768103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.6769665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.7028047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.7029306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.8405984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.8407343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.9739043Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T21:06:43.9740752Z "You've set static_graph to be True, no need to set it again." 2022-03-17T21:06:43.9742901Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T21:06:43.9744443Z "You've set static_graph to be True, no need to set it again." 2022-03-17T21:06:43.9979552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:43.9981027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.1534619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.1536214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.3114202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.3115586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.4611380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.4612752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.4872466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.4873743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.6246943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.6248268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.7756302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.7757591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.8012885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.8014261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.9416702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.9418103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.9696562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:44.9698316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.1173086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.1174497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.2925727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.2927176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.3578570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.3579954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.5226388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:45.5227847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:48.3661108Z ok (16.754s) 2022-03-17T21:06:48.3661963Z 2022-03-17T21:06:48.3663226Z ---------------------------------------------------------------------- 2022-03-17T21:06:48.3664251Z Ran 1 test in 16.755s 2022-03-17T21:06:48.3664737Z 2022-03-17T21:06:48.3664989Z OK 2022-03-17T21:06:48.3665360Z 2022-03-17T21:06:48.3665663Z Generating XML reports... 2022-03-17T21:06:48.3743980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210631.xml 2022-03-17T21:06:51.1446536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:51.1460286Z 2022-03-17T21:06:51.1460509Z Running tests... 2022-03-17T21:06:51.1461216Z ---------------------------------------------------------------------- 2022-03-17T21:06:51.7397058Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23944 2022-03-17T21:06:51.7433076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23945 2022-03-17T21:06:51.7434683Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:52.8963910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:52.8964702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:52.8965936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:52.8966797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:52.9072963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:52.9978931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:53.0389658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:53.0390733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:53.0622316Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:53.0623848Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:53.0649329Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:53.0651236Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:54.8515792Z ok (3.705s) 2022-03-17T21:06:54.8516397Z 2022-03-17T21:06:54.8517619Z ---------------------------------------------------------------------- 2022-03-17T21:06:54.8518641Z Ran 1 test in 3.705s 2022-03-17T21:06:54.8518988Z 2022-03-17T21:06:54.8519252Z OK 2022-03-17T21:06:54.8519803Z 2022-03-17T21:06:54.8520262Z Generating XML reports... 2022-03-17T21:06:54.8585747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210651.xml 2022-03-17T21:06:57.4377941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:57.4390603Z 2022-03-17T21:06:57.4390836Z Running tests... 2022-03-17T21:06:57.4391477Z ---------------------------------------------------------------------- 2022-03-17T21:06:58.0358407Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24220 2022-03-17T21:06:58.0395740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24221 2022-03-17T21:06:58.0397328Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:06:59.1460937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:59.1464635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:59.1465575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:59.1563729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:59.1567963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:59.2483053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:59.2932555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:59.2934202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:59.3222779Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:59.3223750Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:59.3225051Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:59.3225966Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:00.9471140Z ok (3.507s) 2022-03-17T21:07:00.9471828Z 2022-03-17T21:07:00.9473130Z ---------------------------------------------------------------------- 2022-03-17T21:07:00.9473971Z Ran 1 test in 3.508s 2022-03-17T21:07:00.9474314Z 2022-03-17T21:07:00.9474527Z OK 2022-03-17T21:07:00.9474839Z 2022-03-17T21:07:00.9475141Z Generating XML reports... 2022-03-17T21:07:00.9541869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210657.xml 2022-03-17T21:07:03.4961186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:03.4972870Z 2022-03-17T21:07:03.4973101Z Running tests... 2022-03-17T21:07:03.4973732Z ---------------------------------------------------------------------- 2022-03-17T21:07:04.0984483Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24496 2022-03-17T21:07:04.1019223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24497 2022-03-17T21:07:04.1020640Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:05.2471202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:05.2472512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:05.2474364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:05.2476218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:05.2577249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:05.3484010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:15.1904961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.1906264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.2297224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.2298537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.2454597Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:15.2456819Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:15.2460383Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:15.2462428Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:15.3823164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.3824523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.4187208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.4188615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.4572460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.4573760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.4924550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:15.4925861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:18.4352067Z ok (14.937s) 2022-03-17T21:07:18.4352925Z 2022-03-17T21:07:18.4354222Z ---------------------------------------------------------------------- 2022-03-17T21:07:18.4355084Z Ran 1 test in 14.938s 2022-03-17T21:07:18.4355505Z 2022-03-17T21:07:18.4356150Z OK 2022-03-17T21:07:18.4356485Z 2022-03-17T21:07:18.4356757Z Generating XML reports... 2022-03-17T21:07:18.4422563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210703.xml 2022-03-17T21:07:21.2993786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:21.3009119Z 2022-03-17T21:07:21.3009592Z Running tests... 2022-03-17T21:07:21.3010853Z ---------------------------------------------------------------------- 2022-03-17T21:07:21.9155787Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24718 2022-03-17T21:07:21.9197429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24719 2022-03-17T21:07:21.9199888Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:23.0843743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:23.0844359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:23.0845333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:23.0846187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:23.0957251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:23.1858864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:23.5822731Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:23.5824339Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:23.8689001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:23.8690364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:23.8951461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:23.8953790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:23.9059949Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:23.9061162Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:23.9063022Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:23.9064335Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:25.5300599Z ok (4.229s) 2022-03-17T21:07:25.5300780Z 2022-03-17T21:07:25.5301277Z ---------------------------------------------------------------------- 2022-03-17T21:07:25.5304038Z Ran 1 test in 4.229s 2022-03-17T21:07:25.5304432Z 2022-03-17T21:07:25.5304658Z OK 2022-03-17T21:07:25.5304991Z 2022-03-17T21:07:25.5305304Z Generating XML reports... 2022-03-17T21:07:25.5382540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210721.xml 2022-03-17T21:07:28.1700806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:28.1710877Z 2022-03-17T21:07:28.1711594Z Running tests... 2022-03-17T21:07:28.1712636Z ---------------------------------------------------------------------- 2022-03-17T21:07:28.7828025Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24932 2022-03-17T21:07:28.7883780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24933 2022-03-17T21:07:28.7885211Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:29.9546992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:29.9548528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:29.9550838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:29.9552749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:29.9658643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:30.0566006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:30.7472199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:30.7473761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:30.7750531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:30.7752773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:30.7819110Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:30.7821355Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:30.7842274Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:30.7844602Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:32.4005350Z ok (4.229s) 2022-03-17T21:07:32.4010633Z 2022-03-17T21:07:32.4011985Z ---------------------------------------------------------------------- 2022-03-17T21:07:32.4013563Z Ran 1 test in 4.229s 2022-03-17T21:07:32.4014132Z 2022-03-17T21:07:32.4014458Z OK 2022-03-17T21:07:32.4014882Z 2022-03-17T21:07:32.4015349Z Generating XML reports... 2022-03-17T21:07:32.4145972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210728.xml 2022-03-17T21:07:34.9845484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:34.9857730Z 2022-03-17T21:07:34.9858004Z Running tests... 2022-03-17T21:07:34.9859182Z ---------------------------------------------------------------------- 2022-03-17T21:07:35.6232346Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25146 2022-03-17T21:07:35.6274496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25147 2022-03-17T21:07:35.6276268Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:36.7234890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:36.7236148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:36.7238074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:36.7239844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:36.7341596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:36.8250410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:37.3513869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:37.3515081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:39.3391686Z ok (4.353s) 2022-03-17T21:07:39.3392083Z 2022-03-17T21:07:39.3392914Z ---------------------------------------------------------------------- 2022-03-17T21:07:39.3394032Z Ran 1 test in 4.353s 2022-03-17T21:07:39.3394616Z 2022-03-17T21:07:39.3394835Z OK 2022-03-17T21:07:39.3396310Z 2022-03-17T21:07:39.3396683Z Generating XML reports... 2022-03-17T21:07:39.3465875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210734.xml 2022-03-17T21:07:41.9714478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:41.9727108Z 2022-03-17T21:07:41.9727348Z Running tests... 2022-03-17T21:07:41.9727783Z ---------------------------------------------------------------------- 2022-03-17T21:07:42.6082821Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25360 2022-03-17T21:07:42.6179714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25361 2022-03-17T21:07:42.6181494Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:07:43.7546007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:43.7548050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:43.7549329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:43.7550953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:43.7652608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:43.8566139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:53.6362058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:53.6363345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:53.6751701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:53.6752912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:53.6908655Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:53.6911279Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:53.6914042Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:07:53.6915969Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:07:56.4465527Z ok (14.473s) 2022-03-17T21:07:56.4465914Z 2022-03-17T21:07:56.4466793Z ---------------------------------------------------------------------- 2022-03-17T21:07:56.4467559Z Ran 1 test in 14.474s 2022-03-17T21:07:56.4467923Z 2022-03-17T21:07:56.4468111Z OK 2022-03-17T21:07:56.4468424Z 2022-03-17T21:07:56.4468698Z Generating XML reports... 2022-03-17T21:07:56.4549947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210741.xml 2022-03-17T21:07:59.5348366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:59.5358185Z 2022-03-17T21:07:59.5358344Z Running tests... 2022-03-17T21:07:59.5358881Z ---------------------------------------------------------------------- 2022-03-17T21:08:00.1358302Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25582 2022-03-17T21:08:00.1391648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25583 2022-03-17T21:08:00.1393009Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:01.2947450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:01.2948375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:01.2948960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:01.2949710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:01.2963795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:01.3959282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:11.2140882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:11.2145455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:11.2488236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:11.2489772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:11.2615766Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:08:11.2617970Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:08:11.2620948Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:08:11.2623585Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:08:14.0677923Z ok (14.531s) 2022-03-17T21:08:14.0678350Z 2022-03-17T21:08:14.0679226Z ---------------------------------------------------------------------- 2022-03-17T21:08:14.0679997Z Ran 1 test in 14.532s 2022-03-17T21:08:14.0680371Z 2022-03-17T21:08:14.0680539Z OK 2022-03-17T21:08:14.0680826Z 2022-03-17T21:08:14.0681100Z Generating XML reports... 2022-03-17T21:08:14.0761481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210759.xml 2022-03-17T21:08:16.6309474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:16.6322645Z 2022-03-17T21:08:16.6322963Z Running tests... 2022-03-17T21:08:16.6323504Z ---------------------------------------------------------------------- 2022-03-17T21:08:17.2427110Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25796 2022-03-17T21:08:17.2486082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25797 2022-03-17T21:08:17.2488103Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:18.4329460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:18.4331024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:18.4333362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:18.4431784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:18.4448955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:18.5442514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:18.9563106Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:18.9564568Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:19.2523013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:19.2524379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:19.2725592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:19.2726852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:19.2799892Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:08:19.2802228Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:08:19.2830660Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:08:19.2832775Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:08:20.9604634Z ok (4.328s) 2022-03-17T21:08:20.9605534Z 2022-03-17T21:08:20.9607093Z ---------------------------------------------------------------------- 2022-03-17T21:08:20.9607868Z Ran 1 test in 4.328s 2022-03-17T21:08:20.9608259Z 2022-03-17T21:08:20.9608448Z OK 2022-03-17T21:08:20.9608739Z 2022-03-17T21:08:20.9609014Z Generating XML reports... 2022-03-17T21:08:20.9691482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210816.xml 2022-03-17T21:08:23.9988411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:23.9998695Z 2022-03-17T21:08:23.9998990Z Running tests... 2022-03-17T21:08:24.0000146Z ---------------------------------------------------------------------- 2022-03-17T21:08:24.6329807Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26010 2022-03-17T21:08:24.6371659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26011 2022-03-17T21:08:24.6372998Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:25.7428050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:25.7430489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:25.7431840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:25.7433498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:25.7445094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:25.8442231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:27.4447508Z skip (3.444s) 2022-03-17T21:08:27.4448153Z 2022-03-17T21:08:27.4449303Z ---------------------------------------------------------------------- 2022-03-17T21:08:27.4450060Z Ran 1 test in 3.445s 2022-03-17T21:08:27.4450431Z 2022-03-17T21:08:27.4450662Z OK (skipped=1) 2022-03-17T21:08:27.4451007Z 2022-03-17T21:08:27.4451277Z Generating XML reports... 2022-03-17T21:08:27.4535133Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210823.xml 2022-03-17T21:08:30.0986755Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:30.0996782Z 2022-03-17T21:08:30.0997020Z Running tests... 2022-03-17T21:08:30.0997460Z ---------------------------------------------------------------------- 2022-03-17T21:08:30.7328846Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26220 2022-03-17T21:08:30.7424702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26221 2022-03-17T21:08:30.7426160Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:31.8465114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:31.8466155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:31.8466692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:31.8467681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:31.8482298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:31.9481479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:33.7508651Z ok (3.651s) 2022-03-17T21:08:33.7509508Z 2022-03-17T21:08:33.7510639Z ---------------------------------------------------------------------- 2022-03-17T21:08:33.7511448Z Ran 1 test in 3.651s 2022-03-17T21:08:33.7511816Z 2022-03-17T21:08:33.7512004Z OK 2022-03-17T21:08:33.7512300Z 2022-03-17T21:08:33.7512573Z Generating XML reports... 2022-03-17T21:08:33.7586862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210830.xml 2022-03-17T21:08:36.5708793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:36.5721368Z 2022-03-17T21:08:36.5721642Z Running tests... 2022-03-17T21:08:36.5722160Z ---------------------------------------------------------------------- 2022-03-17T21:08:37.1923719Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26430 2022-03-17T21:08:37.1968973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26431 2022-03-17T21:08:37.1970573Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:38.3345499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:38.3346749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:38.3348632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:38.3350379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:38.3460335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:38.4357849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:48.1244142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:48.1245368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:48.2146043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:48.2147283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:08:51.0296339Z ok (14.457s) 2022-03-17T21:08:51.0296751Z 2022-03-17T21:08:51.0297932Z ---------------------------------------------------------------------- 2022-03-17T21:08:51.0298731Z Ran 1 test in 14.458s 2022-03-17T21:08:51.0299090Z 2022-03-17T21:08:51.0299273Z OK 2022-03-17T21:08:51.0299563Z 2022-03-17T21:08:51.0299838Z Generating XML reports... 2022-03-17T21:08:51.0380289Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210836.xml 2022-03-17T21:08:53.9269500Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:53.9281712Z 2022-03-17T21:08:53.9283779Z Running tests... 2022-03-17T21:08:53.9285292Z ---------------------------------------------------------------------- 2022-03-17T21:08:54.5297870Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26644 2022-03-17T21:08:54.5329757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26645 2022-03-17T21:08:54.5331860Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:08:55.6904366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:55.6905952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:55.6908419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:55.6910252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:55.7020465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:55.7924863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:57.8432796Z ok (3.914s) 2022-03-17T21:08:57.8433324Z 2022-03-17T21:08:57.8434497Z ---------------------------------------------------------------------- 2022-03-17T21:08:57.8435488Z Ran 1 test in 3.915s 2022-03-17T21:08:57.8436002Z 2022-03-17T21:08:57.8436226Z OK 2022-03-17T21:08:57.8438786Z 2022-03-17T21:08:57.8439287Z Generating XML reports... 2022-03-17T21:08:57.8506414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210853.xml 2022-03-17T21:09:00.7167631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:00.7176950Z 2022-03-17T21:09:00.7177336Z Running tests... 2022-03-17T21:09:00.7178478Z ---------------------------------------------------------------------- 2022-03-17T21:09:00.7197023Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:09:00.7197718Z 2022-03-17T21:09:00.7198567Z ---------------------------------------------------------------------- 2022-03-17T21:09:00.7199342Z Ran 1 test in 0.002s 2022-03-17T21:09:00.7199713Z 2022-03-17T21:09:00.7199928Z OK (skipped=1) 2022-03-17T21:09:00.7200274Z 2022-03-17T21:09:00.7200542Z Generating XML reports... 2022-03-17T21:09:00.7235425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210900.xml 2022-03-17T21:09:03.2628120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:03.2642730Z 2022-03-17T21:09:03.2644032Z Running tests... 2022-03-17T21:09:03.2645516Z ---------------------------------------------------------------------- 2022-03-17T21:09:03.2660915Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T21:09:03.8748747Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26921 2022-03-17T21:09:03.8848862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26922 2022-03-17T21:09:03.8850241Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:05.0410326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:05.0410881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:05.0411744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:05.0509323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:05.0523343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:05.1523177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:06.8952222Z ok (3.631s) 2022-03-17T21:09:06.8952999Z 2022-03-17T21:09:06.8954245Z ---------------------------------------------------------------------- 2022-03-17T21:09:06.8955062Z Ran 1 test in 3.631s 2022-03-17T21:09:06.8955414Z 2022-03-17T21:09:06.8955625Z OK 2022-03-17T21:09:06.8955917Z 2022-03-17T21:09:06.8956194Z Generating XML reports... 2022-03-17T21:09:06.9021954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210903.xml 2022-03-17T21:09:09.8291470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:09.8304039Z 2022-03-17T21:09:09.8304447Z Running tests... 2022-03-17T21:09:09.8324129Z ---------------------------------------------------------------------- 2022-03-17T21:09:09.8324637Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:09:10.4297102Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27197 2022-03-17T21:09:10.4335883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27198 2022-03-17T21:09:10.4337349Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:11.5556397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:11.5558661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:11.5560089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:11.5561817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:11.5570144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:11.6568350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:11.6928202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:09:11.6929660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:09:13.4452583Z ok (3.615s) 2022-03-17T21:09:13.4453168Z 2022-03-17T21:09:13.4454222Z ---------------------------------------------------------------------- 2022-03-17T21:09:13.4455195Z Ran 1 test in 3.615s 2022-03-17T21:09:13.4455646Z 2022-03-17T21:09:13.4455905Z OK 2022-03-17T21:09:13.4456281Z 2022-03-17T21:09:13.4456627Z Generating XML reports... 2022-03-17T21:09:13.4489195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210909.xml 2022-03-17T21:09:15.9858785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:15.9871356Z 2022-03-17T21:09:15.9871830Z Running tests... 2022-03-17T21:09:15.9873067Z ---------------------------------------------------------------------- 2022-03-17T21:09:15.9898789Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:09:16.5860900Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27473 2022-03-17T21:09:16.5960218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27474 2022-03-17T21:09:16.5961765Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:17.7287172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:17.7288700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:17.7290991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:17.7391135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:17.7395898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:17.8310543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:17.8707454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:09:17.8710250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:09:19.7047714Z ok (3.717s) 2022-03-17T21:09:19.7048274Z 2022-03-17T21:09:19.7049688Z ---------------------------------------------------------------------- 2022-03-17T21:09:19.7050622Z Ran 1 test in 3.717s 2022-03-17T21:09:19.7051105Z 2022-03-17T21:09:19.7051336Z OK 2022-03-17T21:09:19.7051696Z 2022-03-17T21:09:19.7052010Z Generating XML reports... 2022-03-17T21:09:19.7134517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210915.xml 2022-03-17T21:09:22.5929061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:22.5938438Z 2022-03-17T21:09:22.5938710Z Running tests... 2022-03-17T21:09:22.5939293Z ---------------------------------------------------------------------- 2022-03-17T21:09:22.5953887Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:09:23.1878448Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27749 2022-03-17T21:09:23.1911921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27750 2022-03-17T21:09:23.1913245Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:24.3097288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:24.3099524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:24.3100964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:24.3102636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:24.3206527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:24.4117621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:26.1994727Z ok (3.605s) 2022-03-17T21:09:26.1995305Z 2022-03-17T21:09:26.1997040Z ---------------------------------------------------------------------- 2022-03-17T21:09:26.1998188Z Ran 1 test in 3.605s 2022-03-17T21:09:26.1998543Z 2022-03-17T21:09:26.1998727Z OK 2022-03-17T21:09:26.2005626Z 2022-03-17T21:09:26.2006344Z Generating XML reports... 2022-03-17T21:09:26.2065228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210922.xml 2022-03-17T21:09:28.7572724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:28.7583057Z 2022-03-17T21:09:28.7583481Z Running tests... 2022-03-17T21:09:28.7584239Z ---------------------------------------------------------------------- 2022-03-17T21:09:29.3940903Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28025 2022-03-17T21:09:29.3988410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28026 2022-03-17T21:09:29.4013243Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:30.5160990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:30.5161569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:30.5162718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:30.5265307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:30.5270157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:30.6274934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:32.9082738Z ok (4.149s) 2022-03-17T21:09:32.9088330Z 2022-03-17T21:09:32.9090231Z ---------------------------------------------------------------------- 2022-03-17T21:09:32.9091048Z Ran 1 test in 4.150s 2022-03-17T21:09:32.9091415Z 2022-03-17T21:09:32.9091687Z OK 2022-03-17T21:09:32.9092087Z 2022-03-17T21:09:32.9092474Z Generating XML reports... 2022-03-17T21:09:32.9180114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210928.xml 2022-03-17T21:09:35.5644137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:35.5656199Z 2022-03-17T21:09:35.5656644Z Running tests... 2022-03-17T21:09:35.5657176Z ---------------------------------------------------------------------- 2022-03-17T21:09:36.1641660Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28241 2022-03-17T21:09:36.1676458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28242 2022-03-17T21:09:36.1677781Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:37.3133165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:37.3133745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:37.3134616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:37.3135437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:37.3148317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:37.4144316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:39.3765564Z ok (3.810s) 2022-03-17T21:09:39.3766015Z 2022-03-17T21:09:39.3767560Z ---------------------------------------------------------------------- 2022-03-17T21:09:39.3768377Z Ran 1 test in 3.811s 2022-03-17T21:09:39.3768766Z 2022-03-17T21:09:39.3768977Z OK 2022-03-17T21:09:39.3769289Z 2022-03-17T21:09:39.3769587Z Generating XML reports... 2022-03-17T21:09:39.3846265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210935.xml 2022-03-17T21:09:42.1807973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:42.1819150Z 2022-03-17T21:09:42.1819468Z Running tests... 2022-03-17T21:09:42.1819990Z ---------------------------------------------------------------------- 2022-03-17T21:09:42.8142667Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28457 2022-03-17T21:09:42.8208649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28458 2022-03-17T21:09:42.8210011Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:43.9669402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:43.9670626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:43.9671252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:43.9672309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:43.9780818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:44.0693355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:44.0903989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:09:44.0904938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:09:44.0905520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:09:44.0906278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:09:45.9297655Z ok (3.747s) 2022-03-17T21:09:45.9298071Z 2022-03-17T21:09:45.9299102Z ---------------------------------------------------------------------- 2022-03-17T21:09:45.9299908Z Ran 1 test in 3.748s 2022-03-17T21:09:45.9300286Z 2022-03-17T21:09:45.9300473Z OK 2022-03-17T21:09:45.9300748Z 2022-03-17T21:09:45.9301026Z Generating XML reports... 2022-03-17T21:09:45.9385707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210942.xml 2022-03-17T21:09:48.6259021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:48.6271180Z 2022-03-17T21:09:48.6271505Z Running tests... 2022-03-17T21:09:48.6272080Z ---------------------------------------------------------------------- 2022-03-17T21:09:49.2248228Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28679 2022-03-17T21:09:49.2281534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28680 2022-03-17T21:09:49.2282429Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:50.3768377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:50.3770045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:50.3771901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:50.3773692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:50.3884290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:50.4788547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:52.1377030Z skip (3.510s) 2022-03-17T21:09:52.1377463Z 2022-03-17T21:09:52.1378281Z ---------------------------------------------------------------------- 2022-03-17T21:09:52.1379060Z Ran 1 test in 3.510s 2022-03-17T21:09:52.1379425Z 2022-03-17T21:09:52.1379647Z OK (skipped=1) 2022-03-17T21:09:52.1379992Z 2022-03-17T21:09:52.1380257Z Generating XML reports... 2022-03-17T21:09:52.1459200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210948.xml 2022-03-17T21:09:54.7243900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:54.7254409Z 2022-03-17T21:09:54.7254562Z Running tests... 2022-03-17T21:09:54.7255090Z ---------------------------------------------------------------------- 2022-03-17T21:09:55.3357645Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28889 2022-03-17T21:09:55.3405161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28890 2022-03-17T21:09:55.3406911Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:09:56.4806546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:56.4807774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:56.4809654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:56.4811408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:56.4913434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:56.5822687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:58.3502638Z ok (3.624s) 2022-03-17T21:09:58.3503034Z 2022-03-17T21:09:58.3505195Z ---------------------------------------------------------------------- 2022-03-17T21:09:58.3506039Z Ran 1 test in 3.625s 2022-03-17T21:09:58.3506428Z 2022-03-17T21:09:58.3506639Z OK 2022-03-17T21:09:58.3506938Z 2022-03-17T21:09:58.3507223Z Generating XML reports... 2022-03-17T21:09:58.3584818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210954.xml 2022-03-17T21:10:01.0509668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:01.0519313Z 2022-03-17T21:10:01.0519603Z Running tests... 2022-03-17T21:10:01.0520697Z ---------------------------------------------------------------------- 2022-03-17T21:10:01.6536005Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29105 2022-03-17T21:10:01.6581356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29106 2022-03-17T21:10:01.6582640Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:02.7851046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:02.7853058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:02.7855030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:02.7856866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:02.7959747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:02.8863826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:04.7679543Z ok (3.715s) 2022-03-17T21:10:04.7680003Z 2022-03-17T21:10:04.7681034Z ---------------------------------------------------------------------- 2022-03-17T21:10:04.7682374Z Ran 1 test in 3.716s 2022-03-17T21:10:04.7682963Z 2022-03-17T21:10:04.7683179Z OK 2022-03-17T21:10:04.7683470Z 2022-03-17T21:10:04.7683741Z Generating XML reports... 2022-03-17T21:10:04.7762026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211001.xml 2022-03-17T21:10:07.4014131Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:07.4024667Z 2022-03-17T21:10:07.4025060Z Running tests... 2022-03-17T21:10:07.4026412Z ---------------------------------------------------------------------- 2022-03-17T21:10:08.0126118Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29321 2022-03-17T21:10:08.0183659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29322 2022-03-17T21:10:08.0185186Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:09.1748706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:09.1749783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:09.1750439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:09.1751203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:09.1765867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:09.2761298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:11.1277397Z ok (3.725s) 2022-03-17T21:10:11.1277830Z 2022-03-17T21:10:11.1278862Z ---------------------------------------------------------------------- 2022-03-17T21:10:11.1279734Z Ran 1 test in 3.725s 2022-03-17T21:10:11.1280138Z 2022-03-17T21:10:11.1280352Z OK 2022-03-17T21:10:11.1280658Z 2022-03-17T21:10:11.1280980Z Generating XML reports... 2022-03-17T21:10:11.1350517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211007.xml 2022-03-17T21:10:13.7544037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:13.7553109Z 2022-03-17T21:10:13.7553494Z Running tests... 2022-03-17T21:10:13.7554852Z ---------------------------------------------------------------------- 2022-03-17T21:10:13.7571252Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:10:13.7572036Z 2022-03-17T21:10:13.7573050Z ---------------------------------------------------------------------- 2022-03-17T21:10:13.7573958Z Ran 1 test in 0.002s 2022-03-17T21:10:13.7574328Z 2022-03-17T21:10:13.7574557Z OK (skipped=1) 2022-03-17T21:10:13.7574906Z 2022-03-17T21:10:13.7575162Z Generating XML reports... 2022-03-17T21:10:13.7614705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211013.xml 2022-03-17T21:10:16.3181548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:16.3191436Z 2022-03-17T21:10:16.3191614Z Running tests... 2022-03-17T21:10:16.3192414Z ---------------------------------------------------------------------- 2022-03-17T21:10:16.3211229Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:10:16.3211613Z 2022-03-17T21:10:16.3212057Z ---------------------------------------------------------------------- 2022-03-17T21:10:16.3212514Z Ran 1 test in 0.002s 2022-03-17T21:10:16.3212670Z 2022-03-17T21:10:16.3212783Z OK (skipped=1) 2022-03-17T21:10:16.3212941Z 2022-03-17T21:10:16.3213069Z Generating XML reports... 2022-03-17T21:10:16.3251979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211016.xml 2022-03-17T21:10:18.9120872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:18.9130645Z 2022-03-17T21:10:18.9130972Z Running tests... 2022-03-17T21:10:18.9131501Z ---------------------------------------------------------------------- 2022-03-17T21:10:19.5089212Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29671 2022-03-17T21:10:19.5123828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29672 2022-03-17T21:10:19.5125484Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:20.6962195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:20.6963818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:20.6965996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:20.7064434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:20.7084906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:20.8076799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:20.8204885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:20.8206379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:20.8208371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:20.8210117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:22.7200341Z ok (3.806s) 2022-03-17T21:10:22.7200930Z 2022-03-17T21:10:22.7202313Z ---------------------------------------------------------------------- 2022-03-17T21:10:22.7203293Z Ran 1 test in 3.807s 2022-03-17T21:10:22.7203680Z 2022-03-17T21:10:22.7203866Z OK 2022-03-17T21:10:22.7204160Z 2022-03-17T21:10:22.7204438Z Generating XML reports... 2022-03-17T21:10:22.7285538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211018.xml 2022-03-17T21:10:25.6792109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:25.6804153Z 2022-03-17T21:10:25.6804573Z Running tests... 2022-03-17T21:10:25.6805647Z ---------------------------------------------------------------------- 2022-03-17T21:10:26.3124722Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29893 2022-03-17T21:10:26.3169960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29894 2022-03-17T21:10:26.3171307Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:27.5583530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:27.5584986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:27.5587014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:27.5688154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:27.5693234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:27.6698342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:29.3297875Z skip (3.649s) 2022-03-17T21:10:29.3298398Z 2022-03-17T21:10:29.3299433Z ---------------------------------------------------------------------- 2022-03-17T21:10:29.3300313Z Ran 1 test in 3.649s 2022-03-17T21:10:29.3300707Z 2022-03-17T21:10:29.3300975Z OK (skipped=1) 2022-03-17T21:10:29.3301378Z 2022-03-17T21:10:29.3301692Z Generating XML reports... 2022-03-17T21:10:29.3381836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211025.xml 2022-03-17T21:10:32.0386793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:32.0398059Z 2022-03-17T21:10:32.0398448Z Running tests... 2022-03-17T21:10:32.0398987Z ---------------------------------------------------------------------- 2022-03-17T21:10:32.0416732Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:10:32.0417132Z 2022-03-17T21:10:32.0417601Z ---------------------------------------------------------------------- 2022-03-17T21:10:32.0418042Z Ran 1 test in 0.002s 2022-03-17T21:10:32.0418230Z 2022-03-17T21:10:32.0418348Z OK (skipped=1) 2022-03-17T21:10:32.0418502Z 2022-03-17T21:10:32.0418628Z Generating XML reports... 2022-03-17T21:10:32.0456409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211032.xml 2022-03-17T21:10:34.7905358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:34.7912231Z 2022-03-17T21:10:34.7913172Z Running tests... 2022-03-17T21:10:34.7914427Z ---------------------------------------------------------------------- 2022-03-17T21:10:34.7933993Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:10:34.7934787Z 2022-03-17T21:10:34.7935805Z ---------------------------------------------------------------------- 2022-03-17T21:10:34.7936725Z Ran 1 test in 0.002s 2022-03-17T21:10:34.7937068Z 2022-03-17T21:10:34.7937317Z OK (skipped=1) 2022-03-17T21:10:34.7937665Z 2022-03-17T21:10:34.7937938Z Generating XML reports... 2022-03-17T21:10:34.7971846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211034.xml 2022-03-17T21:10:37.2851513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:37.2860113Z 2022-03-17T21:10:37.2860445Z Running tests... 2022-03-17T21:10:37.2861786Z ---------------------------------------------------------------------- 2022-03-17T21:10:37.9007379Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30237 2022-03-17T21:10:37.9052131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30238 2022-03-17T21:10:37.9053656Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:39.0457487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:39.0457964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:39.0458746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:39.0459407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:39.0462314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:39.0463709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:40.7136962Z ok (3.427s) 2022-03-17T21:10:40.7137416Z 2022-03-17T21:10:40.7138671Z ---------------------------------------------------------------------- 2022-03-17T21:10:40.7140568Z Ran 1 test in 3.427s 2022-03-17T21:10:40.7141045Z 2022-03-17T21:10:40.7141270Z OK 2022-03-17T21:10:40.7141563Z 2022-03-17T21:10:40.7141845Z Generating XML reports... 2022-03-17T21:10:40.7221756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211037.xml 2022-03-17T21:10:43.3810605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:43.3820939Z 2022-03-17T21:10:43.3821399Z Running tests... 2022-03-17T21:10:43.3823110Z ---------------------------------------------------------------------- 2022-03-17T21:10:43.9809749Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30447 2022-03-17T21:10:43.9844861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30448 2022-03-17T21:10:43.9846440Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:45.1284351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:45.1285835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:45.1287868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:45.1289860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:45.1291593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:45.1293019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:45.1607596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:45.1609331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:45.1611528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:45.1613597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:45.1878365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:10:45.1880608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:10:45.1882189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:10:45.1883895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:10:45.2116884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T21:10:45.2118402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T21:10:45.2120357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:10:45.2122316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:10:46.9949699Z ok (3.612s) 2022-03-17T21:10:46.9950263Z 2022-03-17T21:10:46.9951810Z ---------------------------------------------------------------------- 2022-03-17T21:10:46.9952744Z Ran 1 test in 3.613s 2022-03-17T21:10:46.9953185Z 2022-03-17T21:10:46.9953442Z OK 2022-03-17T21:10:46.9953811Z 2022-03-17T21:10:46.9954145Z Generating XML reports... 2022-03-17T21:10:47.0027569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211043.xml 2022-03-17T21:10:49.5455241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:49.5466814Z 2022-03-17T21:10:49.5467491Z Running tests... 2022-03-17T21:10:49.5468884Z ---------------------------------------------------------------------- 2022-03-17T21:10:50.1441223Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30681 2022-03-17T21:10:50.1483179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30682 2022-03-17T21:10:50.1485299Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:51.2520071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:51.2520793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:51.2521644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:51.2623952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:51.2629982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:51.3632970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:51.3840154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:51.3840935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:51.3841774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:51.3942724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:53.2588074Z ok (3.712s) 2022-03-17T21:10:53.2588639Z 2022-03-17T21:10:53.2589809Z ---------------------------------------------------------------------- 2022-03-17T21:10:53.2590794Z Ran 1 test in 3.712s 2022-03-17T21:10:53.2591143Z 2022-03-17T21:10:53.2591326Z OK 2022-03-17T21:10:53.2591614Z 2022-03-17T21:10:53.2591904Z Generating XML reports... 2022-03-17T21:10:53.2665766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211049.xml 2022-03-17T21:10:55.8200986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:55.8216093Z 2022-03-17T21:10:55.8216561Z Running tests... 2022-03-17T21:10:55.8217682Z ---------------------------------------------------------------------- 2022-03-17T21:10:56.4473815Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30903 2022-03-17T21:10:56.4522925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30904 2022-03-17T21:10:56.4524311Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:10:57.5905029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:57.5906304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:57.5908212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:57.5910047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:57.6012392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:57.6918994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:57.7124151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:57.7125455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:57.7127310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:57.7129520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:00.1622770Z ok (4.340s) 2022-03-17T21:11:00.1623163Z 2022-03-17T21:11:00.1624634Z ---------------------------------------------------------------------- 2022-03-17T21:11:00.1625477Z Ran 1 test in 4.341s 2022-03-17T21:11:00.1625829Z 2022-03-17T21:11:00.1626016Z OK 2022-03-17T21:11:00.1626329Z 2022-03-17T21:11:00.1626609Z Generating XML reports... 2022-03-17T21:11:00.1695388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211055.xml 2022-03-17T21:11:02.8035248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:02.8046772Z 2022-03-17T21:11:02.8047815Z Running tests... 2022-03-17T21:11:02.8048594Z ---------------------------------------------------------------------- 2022-03-17T21:11:03.4033772Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31125 2022-03-17T21:11:03.4067528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31126 2022-03-17T21:11:03.4069253Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:04.5453709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:04.5454979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:04.5456983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:04.5557256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:04.5558564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:04.6478494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:04.6675428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:11:04.6676658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:11:04.6678875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:04.6777442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:06.6169500Z ok (3.811s) 2022-03-17T21:11:06.6170513Z 2022-03-17T21:11:06.6171961Z ---------------------------------------------------------------------- 2022-03-17T21:11:06.6172921Z Ran 1 test in 3.812s 2022-03-17T21:11:06.6173320Z 2022-03-17T21:11:06.6173552Z OK 2022-03-17T21:11:06.6173878Z 2022-03-17T21:11:06.6174144Z Generating XML reports... 2022-03-17T21:11:06.6250756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211102.xml 2022-03-17T21:11:09.1687583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:09.1701439Z 2022-03-17T21:11:09.1701837Z Running tests... 2022-03-17T21:11:09.1702268Z ---------------------------------------------------------------------- 2022-03-17T21:11:09.7683642Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31347 2022-03-17T21:11:09.7751024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31348 2022-03-17T21:11:09.7752579Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:10.9527358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:10.9529352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:10.9530725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:10.9532501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:10.9633879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:11.0543012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:11.0747392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:11:11.0748675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:11:11.0750571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:11.0752439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:12.8830710Z ok (3.712s) 2022-03-17T21:11:12.8831405Z 2022-03-17T21:11:12.8833249Z ---------------------------------------------------------------------- 2022-03-17T21:11:12.8834084Z Ran 1 test in 3.713s 2022-03-17T21:11:12.8834459Z 2022-03-17T21:11:12.8834629Z OK 2022-03-17T21:11:12.8834927Z 2022-03-17T21:11:12.8835204Z Generating XML reports... 2022-03-17T21:11:12.8908848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211109.xml 2022-03-17T21:11:15.4493668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:15.4504356Z 2022-03-17T21:11:15.4504626Z Running tests... 2022-03-17T21:11:15.4505076Z ---------------------------------------------------------------------- 2022-03-17T21:11:16.0481344Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31569 2022-03-17T21:11:16.0568627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31570 2022-03-17T21:11:16.0571865Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:17.2020461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:17.2021947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:17.2024006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:17.2123938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:17.2126511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:17.3140224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:19.1659545Z skip (3.715s) 2022-03-17T21:11:19.1660075Z 2022-03-17T21:11:19.1661273Z ---------------------------------------------------------------------- 2022-03-17T21:11:19.1662237Z Ran 1 test in 3.715s 2022-03-17T21:11:19.1662684Z 2022-03-17T21:11:19.1662971Z OK (skipped=1) 2022-03-17T21:11:19.1663415Z 2022-03-17T21:11:19.1663767Z Generating XML reports... 2022-03-17T21:11:19.1749387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211115.xml 2022-03-17T21:11:22.0240971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:22.0255570Z 2022-03-17T21:11:22.0255840Z Running tests... 2022-03-17T21:11:22.0256264Z ---------------------------------------------------------------------- 2022-03-17T21:11:22.6383391Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31779 2022-03-17T21:11:22.6435458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31780 2022-03-17T21:11:22.6437236Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:23.8145674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:23.8148063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:23.8149635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:23.8151490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:23.8257569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:23.9169509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:25.5519469Z skip (3.526s) 2022-03-17T21:11:25.5520024Z 2022-03-17T21:11:25.5521122Z ---------------------------------------------------------------------- 2022-03-17T21:11:25.5522445Z Ran 1 test in 3.527s 2022-03-17T21:11:25.5522924Z 2022-03-17T21:11:25.5523243Z OK (skipped=1) 2022-03-17T21:11:25.5523694Z 2022-03-17T21:11:25.5523977Z Generating XML reports... 2022-03-17T21:11:25.5589962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211122.xml 2022-03-17T21:11:28.2142677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:28.2154078Z 2022-03-17T21:11:28.2155066Z Running tests... 2022-03-17T21:11:28.2156488Z ---------------------------------------------------------------------- 2022-03-17T21:11:28.8135036Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31989 2022-03-17T21:11:28.8168115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31990 2022-03-17T21:11:28.8170247Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:29.8917905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:29.8918500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:29.8919445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:29.8920268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:29.9032403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:29.9937237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:31.7292759Z skip (3.513s) 2022-03-17T21:11:31.7293481Z 2022-03-17T21:11:31.7295219Z ---------------------------------------------------------------------- 2022-03-17T21:11:31.7296206Z Ran 1 test in 3.514s 2022-03-17T21:11:31.7296650Z 2022-03-17T21:11:31.7296961Z OK (skipped=1) 2022-03-17T21:11:31.7297404Z 2022-03-17T21:11:31.7297665Z Generating XML reports... 2022-03-17T21:11:31.7377128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211128.xml 2022-03-17T21:11:34.3298182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:34.3310915Z 2022-03-17T21:11:34.3311323Z Running tests... 2022-03-17T21:11:34.3313267Z ---------------------------------------------------------------------- 2022-03-17T21:11:34.9457389Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32199 2022-03-17T21:11:34.9506675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32200 2022-03-17T21:11:34.9508022Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:36.0955211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:36.0956363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:36.0958173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:36.0959946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:36.1067520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:36.1973487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:38.0602750Z skip (3.729s) 2022-03-17T21:11:38.0603548Z 2022-03-17T21:11:38.0604483Z ---------------------------------------------------------------------- 2022-03-17T21:11:38.0605284Z Ran 1 test in 3.729s 2022-03-17T21:11:38.0605629Z 2022-03-17T21:11:38.0605880Z OK (skipped=1) 2022-03-17T21:11:38.0606224Z 2022-03-17T21:11:38.0606493Z Generating XML reports... 2022-03-17T21:11:38.0677862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211134.xml 2022-03-17T21:11:40.7053019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:40.7064374Z 2022-03-17T21:11:40.7064628Z Running tests... 2022-03-17T21:11:40.7065054Z ---------------------------------------------------------------------- 2022-03-17T21:11:41.3051472Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32409 2022-03-17T21:11:41.3085303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32410 2022-03-17T21:11:41.3086681Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:42.4489841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:42.4492060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:42.4493410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:42.4495113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:42.4599551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:42.5503807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:44.4186463Z ok (3.711s) 2022-03-17T21:11:44.4186970Z 2022-03-17T21:11:44.4188818Z ---------------------------------------------------------------------- 2022-03-17T21:11:44.4189873Z Ran 1 test in 3.712s 2022-03-17T21:11:44.4190249Z 2022-03-17T21:11:44.4190462Z OK 2022-03-17T21:11:44.4190876Z 2022-03-17T21:11:44.4191277Z Generating XML reports... 2022-03-17T21:11:44.4265197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211140.xml 2022-03-17T21:11:47.0117982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:47.0130713Z 2022-03-17T21:11:47.0130898Z Running tests... 2022-03-17T21:11:47.0131602Z ---------------------------------------------------------------------- 2022-03-17T21:11:47.6108769Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32625 2022-03-17T21:11:47.6141813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32626 2022-03-17T21:11:47.6143163Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:48.7444557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:48.7446844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:48.7448257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:48.7449926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:48.7555149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:48.8458630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:50.5234340Z ok (3.510s) 2022-03-17T21:11:50.5234933Z 2022-03-17T21:11:50.5236204Z ---------------------------------------------------------------------- 2022-03-17T21:11:50.5237191Z Ran 1 test in 3.510s 2022-03-17T21:11:50.5237665Z 2022-03-17T21:11:50.5237933Z OK 2022-03-17T21:11:50.5238281Z 2022-03-17T21:11:50.5238653Z Generating XML reports... 2022-03-17T21:11:50.5306409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211147.xml 2022-03-17T21:11:53.2970948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:53.2983100Z 2022-03-17T21:11:53.2983327Z Running tests... 2022-03-17T21:11:53.2983857Z ---------------------------------------------------------------------- 2022-03-17T21:11:53.8971366Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32835 2022-03-17T21:11:53.9020540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32836 2022-03-17T21:11:53.9021870Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:11:55.0557340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:55.0558826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:55.0560686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:55.0562620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:55.0671403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:55.1571125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:57.0102394Z ok (3.711s) 2022-03-17T21:11:57.0102805Z 2022-03-17T21:11:57.0103700Z ---------------------------------------------------------------------- 2022-03-17T21:11:57.0104485Z Ran 1 test in 3.712s 2022-03-17T21:11:57.0104840Z 2022-03-17T21:11:57.0105036Z OK 2022-03-17T21:11:57.0105328Z 2022-03-17T21:11:57.0105606Z Generating XML reports... 2022-03-17T21:11:57.0186491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211153.xml 2022-03-17T21:11:59.6100464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:59.6114488Z 2022-03-17T21:11:59.6114812Z Running tests... 2022-03-17T21:11:59.6116247Z ---------------------------------------------------------------------- 2022-03-17T21:12:00.2149590Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33051 2022-03-17T21:12:00.2189141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33052 2022-03-17T21:12:00.2190436Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:01.3232341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:01.3235238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:01.3237309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:01.3239938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:01.3252929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:01.4242610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:03.2274793Z ok (3.616s) 2022-03-17T21:12:03.2275314Z 2022-03-17T21:12:03.2276676Z ---------------------------------------------------------------------- 2022-03-17T21:12:03.2277608Z Ran 1 test in 3.616s 2022-03-17T21:12:03.2278090Z 2022-03-17T21:12:03.2278335Z OK 2022-03-17T21:12:03.2278643Z 2022-03-17T21:12:03.2278966Z Generating XML reports... 2022-03-17T21:12:03.2347562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211159.xml 2022-03-17T21:12:05.7898908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:05.7910529Z 2022-03-17T21:12:05.7911074Z Running tests... 2022-03-17T21:12:05.7912126Z ---------------------------------------------------------------------- 2022-03-17T21:12:06.4234079Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33267 2022-03-17T21:12:06.4282552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33268 2022-03-17T21:12:06.4284082Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:07.5420582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:07.5422056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:07.5424546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:07.5426743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:07.5428254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:07.5429381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:09.4363256Z ok (3.645s) 2022-03-17T21:12:09.4363936Z 2022-03-17T21:12:09.4365068Z ---------------------------------------------------------------------- 2022-03-17T21:12:09.4366006Z Ran 1 test in 3.645s 2022-03-17T21:12:09.4366446Z 2022-03-17T21:12:09.4366715Z OK 2022-03-17T21:12:09.4367016Z 2022-03-17T21:12:09.4367289Z Generating XML reports... 2022-03-17T21:12:09.4447090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211205.xml 2022-03-17T21:12:12.0408842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:12.0421016Z 2022-03-17T21:12:12.0421440Z Running tests... 2022-03-17T21:12:12.0421971Z ---------------------------------------------------------------------- 2022-03-17T21:12:12.6408346Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33483 2022-03-17T21:12:12.6449017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33484 2022-03-17T21:12:12.6450322Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:13.7822827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:13.7824137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:13.7826123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:13.7828084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:13.7829535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:13.7830781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:15.5546769Z ok (3.512s) 2022-03-17T21:12:15.5547238Z 2022-03-17T21:12:15.5548235Z ---------------------------------------------------------------------- 2022-03-17T21:12:15.5549161Z Ran 1 test in 3.512s 2022-03-17T21:12:15.5549603Z 2022-03-17T21:12:15.5549834Z OK 2022-03-17T21:12:15.5550201Z 2022-03-17T21:12:15.5550518Z Generating XML reports... 2022-03-17T21:12:15.5623359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211212.xml 2022-03-17T21:12:18.1122688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:18.1139490Z 2022-03-17T21:12:18.1139737Z Running tests... 2022-03-17T21:12:18.1140228Z ---------------------------------------------------------------------- 2022-03-17T21:12:18.7515956Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33693 2022-03-17T21:12:18.7567681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33694 2022-03-17T21:12:18.7568976Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:19.9137129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:19.9138236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:19.9138847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:19.9139606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:19.9246146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:20.0163580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:20.0375082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:12:20.0376595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:12:20.0378612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:20.0380911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:22.5733290Z ok (4.459s) 2022-03-17T21:12:22.5733841Z 2022-03-17T21:12:22.5734887Z ---------------------------------------------------------------------- 2022-03-17T21:12:22.5735846Z Ran 1 test in 4.460s 2022-03-17T21:12:22.5736322Z 2022-03-17T21:12:22.5736539Z OK 2022-03-17T21:12:22.5736843Z 2022-03-17T21:12:22.5737124Z Generating XML reports... 2022-03-17T21:12:22.5813291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211218.xml 2022-03-17T21:12:25.1674357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:25.1688103Z 2022-03-17T21:12:25.1688391Z Running tests... 2022-03-17T21:12:25.1689844Z ---------------------------------------------------------------------- 2022-03-17T21:12:25.7669835Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33915 2022-03-17T21:12:25.7704768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33916 2022-03-17T21:12:25.7706077Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:26.9169282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:26.9171161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:26.9172457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:26.9174112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:26.9282612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:27.0188662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:27.0399896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:12:27.0401117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:12:27.0403615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:27.0405379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:29.0817502Z ok (3.913s) 2022-03-17T21:12:29.0817869Z 2022-03-17T21:12:29.0818729Z ---------------------------------------------------------------------- 2022-03-17T21:12:29.0819506Z Ran 1 test in 3.913s 2022-03-17T21:12:29.0819869Z 2022-03-17T21:12:29.0820060Z OK 2022-03-17T21:12:29.0820351Z 2022-03-17T21:12:29.0820635Z Generating XML reports... 2022-03-17T21:12:29.0900835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211225.xml 2022-03-17T21:12:31.8938938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:31.8947417Z 2022-03-17T21:12:31.8947703Z Running tests... 2022-03-17T21:12:31.8948778Z ---------------------------------------------------------------------- 2022-03-17T21:12:32.5324221Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34137 2022-03-17T21:12:32.5365393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34138 2022-03-17T21:12:32.5366705Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:33.6273093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:33.6274336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:33.6276148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:33.6277905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:33.6384566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:33.7287509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:33.7298191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:12:33.7393716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:12:33.7395548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:33.7399668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:35.5491046Z ok (3.654s) 2022-03-17T21:12:35.5492187Z 2022-03-17T21:12:35.5493581Z ---------------------------------------------------------------------- 2022-03-17T21:12:35.5494490Z Ran 1 test in 3.654s 2022-03-17T21:12:35.5494862Z 2022-03-17T21:12:35.5495039Z OK 2022-03-17T21:12:35.5495342Z 2022-03-17T21:12:35.5495630Z Generating XML reports... 2022-03-17T21:12:35.5532853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211231.xml 2022-03-17T21:12:38.5526411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:38.5539539Z 2022-03-17T21:12:38.5540216Z Running tests... 2022-03-17T21:12:38.5541676Z ---------------------------------------------------------------------- 2022-03-17T21:12:39.1670943Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34359 2022-03-17T21:12:39.1716683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34360 2022-03-17T21:12:39.1718901Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:40.3233153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:40.3234479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:40.3235071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:40.3235854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:40.3343844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:40.4252178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:40.4463973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:12:40.4465943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:40.4467251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:12:40.4468874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:42.2806495Z ok (3.726s) 2022-03-17T21:12:42.2807710Z 2022-03-17T21:12:42.2808789Z ---------------------------------------------------------------------- 2022-03-17T21:12:42.2809541Z Ran 1 test in 3.726s 2022-03-17T21:12:42.2809904Z 2022-03-17T21:12:42.2810088Z OK 2022-03-17T21:12:42.2810377Z 2022-03-17T21:12:42.2810639Z Generating XML reports... 2022-03-17T21:12:42.2887093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211238.xml 2022-03-17T21:12:44.9269114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:44.9281912Z 2022-03-17T21:12:44.9285599Z Running tests... 2022-03-17T21:12:44.9286168Z ---------------------------------------------------------------------- 2022-03-17T21:12:45.5285820Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34581 2022-03-17T21:12:45.5325092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34582 2022-03-17T21:12:45.5326419Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:46.6504624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:46.6505233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:46.6506349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:46.6507145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:46.6615684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:46.7518916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:48.4421702Z skip (3.514s) 2022-03-17T21:12:48.4422109Z 2022-03-17T21:12:48.4424467Z ---------------------------------------------------------------------- 2022-03-17T21:12:48.4425340Z Ran 1 test in 3.514s 2022-03-17T21:12:48.4425712Z 2022-03-17T21:12:48.4425944Z OK (skipped=1) 2022-03-17T21:12:48.4426319Z 2022-03-17T21:12:48.4426589Z Generating XML reports... 2022-03-17T21:12:48.4505977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211244.xml 2022-03-17T21:12:51.1257998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:51.1270556Z 2022-03-17T21:12:51.1270915Z Running tests... 2022-03-17T21:12:51.1271453Z ---------------------------------------------------------------------- 2022-03-17T21:12:51.7573194Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34791 2022-03-17T21:12:51.7622050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34792 2022-03-17T21:12:51.7623644Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:52.9305644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:52.9307858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:52.9309188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:52.9310840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:52.9321444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:53.0321050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:54.7760824Z skip (3.648s) 2022-03-17T21:12:54.7761508Z 2022-03-17T21:12:54.7763074Z ---------------------------------------------------------------------- 2022-03-17T21:12:54.7764148Z Ran 1 test in 3.649s 2022-03-17T21:12:54.7764518Z 2022-03-17T21:12:54.7764751Z OK (skipped=1) 2022-03-17T21:12:54.7765115Z 2022-03-17T21:12:54.7765384Z Generating XML reports... 2022-03-17T21:12:54.7848183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211251.xml 2022-03-17T21:12:57.5131636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:57.5145083Z 2022-03-17T21:12:57.5145563Z Running tests... 2022-03-17T21:12:57.5146723Z ---------------------------------------------------------------------- 2022-03-17T21:12:58.1480771Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35001 2022-03-17T21:12:58.1523563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35002 2022-03-17T21:12:58.1524909Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:12:59.3034940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:59.3038269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:59.3040546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:59.3136574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:59.3141727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:59.4062165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:01.2611541Z skip (3.746s) 2022-03-17T21:13:01.2611946Z 2022-03-17T21:13:01.2613282Z ---------------------------------------------------------------------- 2022-03-17T21:13:01.2614106Z Ran 1 test in 3.747s 2022-03-17T21:13:01.2614476Z 2022-03-17T21:13:01.2614712Z OK (skipped=1) 2022-03-17T21:13:01.2615063Z 2022-03-17T21:13:01.2615342Z Generating XML reports... 2022-03-17T21:13:01.2699804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211257.xml 2022-03-17T21:13:03.8857957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:03.8870969Z 2022-03-17T21:13:03.8871235Z Running tests... 2022-03-17T21:13:03.8871774Z ---------------------------------------------------------------------- 2022-03-17T21:13:04.5045546Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35211 2022-03-17T21:13:04.5091934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35212 2022-03-17T21:13:04.5093289Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:05.6544523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:05.6545011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:05.6545883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:05.6646337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:05.6657053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:05.7658777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:07.5172172Z skip (3.630s) 2022-03-17T21:13:07.5172763Z 2022-03-17T21:13:07.5174100Z ---------------------------------------------------------------------- 2022-03-17T21:13:07.5174875Z Ran 1 test in 3.630s 2022-03-17T21:13:07.5175245Z 2022-03-17T21:13:07.5175472Z OK (skipped=1) 2022-03-17T21:13:07.5176507Z 2022-03-17T21:13:07.5176764Z Generating XML reports... 2022-03-17T21:13:07.5260373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211303.xml 2022-03-17T21:13:10.1887011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:10.1898729Z 2022-03-17T21:13:10.1899074Z Running tests... 2022-03-17T21:13:10.1899542Z ---------------------------------------------------------------------- 2022-03-17T21:13:10.7889315Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35421 2022-03-17T21:13:10.7927010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35422 2022-03-17T21:13:10.7928541Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:11.9369388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:11.9370632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:11.9372421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:11.9472087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:11.9482811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:12.0483372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:13.9016977Z ok (3.711s) 2022-03-17T21:13:13.9017586Z 2022-03-17T21:13:13.9018677Z ---------------------------------------------------------------------- 2022-03-17T21:13:13.9019660Z Ran 1 test in 3.712s 2022-03-17T21:13:13.9020109Z 2022-03-17T21:13:13.9020365Z OK 2022-03-17T21:13:13.9020706Z 2022-03-17T21:13:13.9021052Z Generating XML reports... 2022-03-17T21:13:13.9097814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211310.xml 2022-03-17T21:13:16.8276107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:16.8287801Z 2022-03-17T21:13:16.8288138Z Running tests... 2022-03-17T21:13:16.8288572Z ---------------------------------------------------------------------- 2022-03-17T21:13:17.4263257Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35637 2022-03-17T21:13:17.4292850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35638 2022-03-17T21:13:17.4294348Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:18.5791301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:18.5792896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:18.5794731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:18.5796505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:18.5897748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:18.6805892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:20.4389627Z ok (3.610s) 2022-03-17T21:13:20.4390071Z 2022-03-17T21:13:20.4391176Z ---------------------------------------------------------------------- 2022-03-17T21:13:20.4392131Z Ran 1 test in 3.610s 2022-03-17T21:13:20.4392573Z 2022-03-17T21:13:20.4392786Z OK 2022-03-17T21:13:20.4398896Z 2022-03-17T21:13:20.4400275Z Generating XML reports... 2022-03-17T21:13:20.4466376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211316.xml 2022-03-17T21:13:23.1206046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:23.1218988Z 2022-03-17T21:13:23.1219378Z Running tests... 2022-03-17T21:13:23.1220580Z ---------------------------------------------------------------------- 2022-03-17T21:13:23.7171249Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35853 2022-03-17T21:13:23.7207844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35854 2022-03-17T21:13:23.7209199Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:24.8726082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:24.8727105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:24.8727710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:24.8728459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:24.8838789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:24.9744630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:27.1302423Z ok (4.008s) 2022-03-17T21:13:27.1302839Z 2022-03-17T21:13:27.1303927Z ---------------------------------------------------------------------- 2022-03-17T21:13:27.1304718Z Ran 1 test in 4.008s 2022-03-17T21:13:27.1305080Z 2022-03-17T21:13:27.1305275Z OK 2022-03-17T21:13:27.1305563Z 2022-03-17T21:13:27.1305852Z Generating XML reports... 2022-03-17T21:13:27.1378569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211323.xml 2022-03-17T21:13:29.7172590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:29.7182582Z 2022-03-17T21:13:29.7183056Z Running tests... 2022-03-17T21:13:29.7184209Z ---------------------------------------------------------------------- 2022-03-17T21:13:30.3561756Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36069 2022-03-17T21:13:30.3606893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36070 2022-03-17T21:13:30.3608188Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:31.5118772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:31.5119790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:31.5120379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:31.5122258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:31.5227480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:31.6134749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:33.6705901Z ok (3.952s) 2022-03-17T21:13:33.6706383Z 2022-03-17T21:13:33.6707445Z ---------------------------------------------------------------------- 2022-03-17T21:13:33.6708232Z Ran 1 test in 3.952s 2022-03-17T21:13:33.6708606Z 2022-03-17T21:13:33.6708801Z OK 2022-03-17T21:13:33.6709082Z 2022-03-17T21:13:33.6709422Z Generating XML reports... 2022-03-17T21:13:33.6786458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211329.xml 2022-03-17T21:13:36.2776167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:36.2783780Z 2022-03-17T21:13:36.2784319Z Running tests... 2022-03-17T21:13:36.2785674Z ---------------------------------------------------------------------- 2022-03-17T21:13:36.9134753Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36285 2022-03-17T21:13:36.9181887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36286 2022-03-17T21:13:36.9183253Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:38.1217804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:38.1218831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:38.1219422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:38.1220322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:38.1234531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:38.2231670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:40.2275879Z ok (3.949s) 2022-03-17T21:13:40.2276244Z 2022-03-17T21:13:40.2277199Z ---------------------------------------------------------------------- 2022-03-17T21:13:40.2277965Z Ran 1 test in 3.949s 2022-03-17T21:13:40.2278341Z 2022-03-17T21:13:40.2278509Z OK 2022-03-17T21:13:40.2278796Z 2022-03-17T21:13:40.2279576Z Generating XML reports... 2022-03-17T21:13:40.2347018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211336.xml 2022-03-17T21:13:42.9268621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:42.9281456Z 2022-03-17T21:13:42.9282185Z Running tests... 2022-03-17T21:13:42.9283386Z ---------------------------------------------------------------------- 2022-03-17T21:13:43.5509500Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36501 2022-03-17T21:13:43.5547345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36502 2022-03-17T21:13:43.5548667Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:44.6966993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:44.6968295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:44.6970338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:44.6972175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:44.6982715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:44.6983899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:47.3646369Z ok (4.436s) 2022-03-17T21:13:47.3646985Z 2022-03-17T21:13:47.3648245Z ---------------------------------------------------------------------- 2022-03-17T21:13:47.3649217Z Ran 1 test in 4.436s 2022-03-17T21:13:47.3649657Z 2022-03-17T21:13:47.3649865Z OK 2022-03-17T21:13:47.3650241Z 2022-03-17T21:13:47.3650592Z Generating XML reports... 2022-03-17T21:13:47.3732259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211342.xml 2022-03-17T21:13:50.0520068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:50.0531969Z 2022-03-17T21:13:50.0532181Z Running tests... 2022-03-17T21:13:50.0532696Z ---------------------------------------------------------------------- 2022-03-17T21:13:50.6825471Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36711 2022-03-17T21:13:50.6875345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36712 2022-03-17T21:13:50.6877089Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:51.8359934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:51.8360993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:51.8363809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:51.8464032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:51.8477536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:51.9474029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:53.9968362Z ok (3.943s) 2022-03-17T21:13:53.9968973Z 2022-03-17T21:13:53.9970319Z ---------------------------------------------------------------------- 2022-03-17T21:13:53.9971199Z Ran 1 test in 3.943s 2022-03-17T21:13:53.9971569Z 2022-03-17T21:13:53.9971753Z OK 2022-03-17T21:13:53.9972539Z 2022-03-17T21:13:53.9972831Z Generating XML reports... 2022-03-17T21:13:54.0055459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211350.xml 2022-03-17T21:13:56.7877112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:56.7891612Z 2022-03-17T21:13:56.7892005Z Running tests... 2022-03-17T21:13:56.7892545Z ---------------------------------------------------------------------- 2022-03-17T21:13:57.3989527Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36927 2022-03-17T21:13:57.4036059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36928 2022-03-17T21:13:57.4037509Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:13:58.5472226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:58.5473079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:58.5473580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:58.5474214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:58.5582218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:58.6491728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:00.4137094Z ok (3.624s) 2022-03-17T21:14:00.4137502Z 2022-03-17T21:14:00.4138436Z ---------------------------------------------------------------------- 2022-03-17T21:14:00.4139253Z Ran 1 test in 3.624s 2022-03-17T21:14:00.4139833Z 2022-03-17T21:14:00.4140174Z OK 2022-03-17T21:14:00.4140471Z 2022-03-17T21:14:00.4140751Z Generating XML reports... 2022-03-17T21:14:00.4222761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211356.xml 2022-03-17T21:14:03.0770318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:03.0779721Z 2022-03-17T21:14:03.0779990Z Running tests... 2022-03-17T21:14:03.0780529Z ---------------------------------------------------------------------- 2022-03-17T21:14:03.7125390Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37143 2022-03-17T21:14:03.7168360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37144 2022-03-17T21:14:03.7169296Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:14:04.8801234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:04.8802389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:04.8802987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:04.8803732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:04.8911641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:04.9820679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:07.3265801Z ok (4.248s) 2022-03-17T21:14:07.3266181Z 2022-03-17T21:14:07.3267162Z ---------------------------------------------------------------------- 2022-03-17T21:14:07.3267925Z Ran 1 test in 4.248s 2022-03-17T21:14:07.3268729Z 2022-03-17T21:14:07.3268925Z OK 2022-03-17T21:14:07.3269226Z 2022-03-17T21:14:07.3269508Z Generating XML reports... 2022-03-17T21:14:07.3346007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211403.xml 2022-03-17T21:14:10.0016927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:10.0031668Z 2022-03-17T21:14:10.0032508Z Running tests... 2022-03-17T21:14:10.0033693Z ---------------------------------------------------------------------- 2022-03-17T21:14:10.6359485Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37359 2022-03-17T21:14:10.6399319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37360 2022-03-17T21:14:10.6400652Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:14:11.7491529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:11.7491998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:11.7493667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:11.7595237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:11.7601123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:11.8517218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:14.7505228Z ok (4.747s) 2022-03-17T21:14:14.7505597Z 2022-03-17T21:14:14.7506477Z ---------------------------------------------------------------------- 2022-03-17T21:14:14.7507257Z Ran 1 test in 4.747s 2022-03-17T21:14:14.7507645Z 2022-03-17T21:14:14.7507855Z OK 2022-03-17T21:14:14.7508138Z 2022-03-17T21:14:14.7508414Z Generating XML reports... 2022-03-17T21:14:14.7586885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211409.xml 2022-03-17T21:14:17.4282898Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:17.4297183Z 2022-03-17T21:14:17.4297288Z Running tests... 2022-03-17T21:14:17.4299699Z ---------------------------------------------------------------------- 2022-03-17T21:14:18.0672777Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37575 2022-03-17T21:14:18.0726187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37576 2022-03-17T21:14:18.0727632Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:14:19.1849738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:19.1851033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:19.1852901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:19.1854753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:19.1866794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:19.2862862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:21.7838246Z ok (4.353s) 2022-03-17T21:14:21.7838628Z 2022-03-17T21:14:21.7839439Z ---------------------------------------------------------------------- 2022-03-17T21:14:21.7840678Z Ran 1 test in 4.354s 2022-03-17T21:14:21.7841056Z 2022-03-17T21:14:21.7841240Z OK 2022-03-17T21:14:21.7841529Z 2022-03-17T21:14:21.7841804Z Generating XML reports... 2022-03-17T21:14:21.7922660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211417.xml 2022-03-17T21:14:24.4301275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:24.4314863Z 2022-03-17T21:14:24.4315236Z Running tests... 2022-03-17T21:14:24.4315672Z ---------------------------------------------------------------------- 2022-03-17T21:14:25.0298930Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37791 2022-03-17T21:14:25.0338414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37792 2022-03-17T21:14:25.0339751Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:14:26.1486090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:26.1487647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:26.1489522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:26.1491365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:26.1592478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:26.2500261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:29.1443990Z ok (4.713s) 2022-03-17T21:14:29.1444387Z 2022-03-17T21:14:29.1445238Z ---------------------------------------------------------------------- 2022-03-17T21:14:29.1446056Z Ran 1 test in 4.713s 2022-03-17T21:14:29.1446439Z 2022-03-17T21:14:29.1446620Z OK 2022-03-17T21:14:29.1446927Z 2022-03-17T21:14:29.1448092Z Generating XML reports... 2022-03-17T21:14:29.1516248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211424.xml 2022-03-17T21:14:31.6870779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:31.6882891Z 2022-03-17T21:14:31.6883166Z Running tests... 2022-03-17T21:14:31.6883664Z ---------------------------------------------------------------------- 2022-03-17T21:14:31.6903351Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:31.6903589Z 2022-03-17T21:14:31.6903973Z ---------------------------------------------------------------------- 2022-03-17T21:14:31.6904413Z Ran 1 test in 0.002s 2022-03-17T21:14:31.6904552Z 2022-03-17T21:14:31.6904650Z OK (skipped=1) 2022-03-17T21:14:31.6904791Z 2022-03-17T21:14:31.6904924Z Generating XML reports... 2022-03-17T21:14:31.6946434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211431.xml 2022-03-17T21:14:34.1825057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:34.1839073Z 2022-03-17T21:14:34.1839602Z Running tests... 2022-03-17T21:14:34.1840752Z ---------------------------------------------------------------------- 2022-03-17T21:14:34.1854826Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:34.1855482Z 2022-03-17T21:14:34.1856278Z ---------------------------------------------------------------------- 2022-03-17T21:14:34.1857039Z Ran 1 test in 0.002s 2022-03-17T21:14:34.1857407Z 2022-03-17T21:14:34.1857641Z OK (skipped=1) 2022-03-17T21:14:34.1857993Z 2022-03-17T21:14:34.1858253Z Generating XML reports... 2022-03-17T21:14:34.1898573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211434.xml 2022-03-17T21:14:36.6811021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:36.6826691Z 2022-03-17T21:14:36.6826915Z Running tests... 2022-03-17T21:14:36.6827339Z ---------------------------------------------------------------------- 2022-03-17T21:14:36.6846052Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:36.6846308Z 2022-03-17T21:14:36.6846601Z ---------------------------------------------------------------------- 2022-03-17T21:14:36.6846883Z Ran 1 test in 0.002s 2022-03-17T21:14:36.6847019Z 2022-03-17T21:14:36.6847106Z OK (skipped=1) 2022-03-17T21:14:36.6847242Z 2022-03-17T21:14:36.6847338Z Generating XML reports... 2022-03-17T21:14:36.6886377Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211436.xml 2022-03-17T21:14:39.2620148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:39.2629149Z 2022-03-17T21:14:39.2629375Z Running tests... 2022-03-17T21:14:39.2629816Z ---------------------------------------------------------------------- 2022-03-17T21:14:39.2647267Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:39.2647525Z 2022-03-17T21:14:39.2647966Z ---------------------------------------------------------------------- 2022-03-17T21:14:39.2648518Z Ran 1 test in 0.002s 2022-03-17T21:14:39.2648657Z 2022-03-17T21:14:39.2648744Z OK (skipped=1) 2022-03-17T21:14:39.2648901Z 2022-03-17T21:14:39.2649022Z Generating XML reports... 2022-03-17T21:14:39.2690960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211439.xml 2022-03-17T21:14:42.0496651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:42.0515263Z 2022-03-17T21:14:42.0515512Z Running tests... 2022-03-17T21:14:42.0515943Z ---------------------------------------------------------------------- 2022-03-17T21:14:42.0532498Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:42.0532748Z 2022-03-17T21:14:42.0533049Z ---------------------------------------------------------------------- 2022-03-17T21:14:42.0533329Z Ran 1 test in 0.002s 2022-03-17T21:14:42.0533469Z 2022-03-17T21:14:42.0533556Z OK (skipped=1) 2022-03-17T21:14:42.0533695Z 2022-03-17T21:14:42.0533800Z Generating XML reports... 2022-03-17T21:14:42.0573075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211442.xml 2022-03-17T21:14:44.6012533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:44.6025620Z 2022-03-17T21:14:44.6026144Z Running tests... 2022-03-17T21:14:44.6027331Z ---------------------------------------------------------------------- 2022-03-17T21:14:44.6043071Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:44.6043766Z 2022-03-17T21:14:44.6044558Z ---------------------------------------------------------------------- 2022-03-17T21:14:44.6045334Z Ran 1 test in 0.002s 2022-03-17T21:14:44.6045704Z 2022-03-17T21:14:44.6045919Z OK (skipped=1) 2022-03-17T21:14:44.6046267Z 2022-03-17T21:14:44.6046538Z Generating XML reports... 2022-03-17T21:14:44.6085924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211444.xml 2022-03-17T21:14:47.0788369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:47.0802773Z 2022-03-17T21:14:47.0803071Z Running tests... 2022-03-17T21:14:47.0803514Z ---------------------------------------------------------------------- 2022-03-17T21:14:47.0820962Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:47.0821212Z 2022-03-17T21:14:47.0821512Z ---------------------------------------------------------------------- 2022-03-17T21:14:47.0821806Z Ran 1 test in 0.002s 2022-03-17T21:14:47.0821937Z 2022-03-17T21:14:47.0822023Z OK (skipped=1) 2022-03-17T21:14:47.0822154Z 2022-03-17T21:14:47.0822257Z Generating XML reports... 2022-03-17T21:14:47.0861648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211447.xml 2022-03-17T21:14:49.5850051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:49.5862953Z 2022-03-17T21:14:49.5863221Z Running tests... 2022-03-17T21:14:49.5863669Z ---------------------------------------------------------------------- 2022-03-17T21:14:49.5880792Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:49.5881326Z 2022-03-17T21:14:49.5882001Z ---------------------------------------------------------------------- 2022-03-17T21:14:49.5882302Z Ran 1 test in 0.002s 2022-03-17T21:14:49.5882442Z 2022-03-17T21:14:49.5882559Z OK (skipped=1) 2022-03-17T21:14:49.5882727Z 2022-03-17T21:14:49.5882856Z Generating XML reports... 2022-03-17T21:14:49.5931528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211449.xml 2022-03-17T21:14:52.1635533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:52.1650920Z 2022-03-17T21:14:52.1651189Z Running tests... 2022-03-17T21:14:52.1651628Z ---------------------------------------------------------------------- 2022-03-17T21:14:52.1668560Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:52.1668825Z 2022-03-17T21:14:52.1669117Z ---------------------------------------------------------------------- 2022-03-17T21:14:52.1669408Z Ran 1 test in 0.002s 2022-03-17T21:14:52.1669536Z 2022-03-17T21:14:52.1669651Z OK (skipped=1) 2022-03-17T21:14:52.1669823Z 2022-03-17T21:14:52.1669943Z Generating XML reports... 2022-03-17T21:14:52.1709212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211452.xml 2022-03-17T21:14:54.6673261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:54.6686093Z 2022-03-17T21:14:54.6686393Z Running tests... 2022-03-17T21:14:54.6686821Z ---------------------------------------------------------------------- 2022-03-17T21:14:54.6706283Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:54.6706555Z 2022-03-17T21:14:54.6706846Z ---------------------------------------------------------------------- 2022-03-17T21:14:54.6707130Z Ran 1 test in 0.002s 2022-03-17T21:14:54.6707269Z 2022-03-17T21:14:54.6707386Z OK (skipped=1) 2022-03-17T21:14:54.6707543Z 2022-03-17T21:14:54.6707673Z Generating XML reports... 2022-03-17T21:14:54.6745948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211454.xml 2022-03-17T21:14:57.1830882Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:57.1844523Z 2022-03-17T21:14:57.1845068Z Running tests... 2022-03-17T21:14:57.1846246Z ---------------------------------------------------------------------- 2022-03-17T21:14:57.1863031Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:57.1863782Z 2022-03-17T21:14:57.1864574Z ---------------------------------------------------------------------- 2022-03-17T21:14:57.1865358Z Ran 1 test in 0.002s 2022-03-17T21:14:57.1865736Z 2022-03-17T21:14:57.1865948Z OK (skipped=1) 2022-03-17T21:14:57.1866297Z 2022-03-17T21:14:57.1866567Z Generating XML reports... 2022-03-17T21:14:57.1904166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211457.xml 2022-03-17T21:14:59.6617586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:59.6630243Z 2022-03-17T21:14:59.6630401Z Running tests... 2022-03-17T21:14:59.6630936Z ---------------------------------------------------------------------- 2022-03-17T21:14:59.6647691Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:59.6647999Z 2022-03-17T21:14:59.6648337Z ---------------------------------------------------------------------- 2022-03-17T21:14:59.6648627Z Ran 1 test in 0.002s 2022-03-17T21:14:59.6648767Z 2022-03-17T21:14:59.6648856Z OK (skipped=1) 2022-03-17T21:14:59.6648989Z 2022-03-17T21:14:59.6649097Z Generating XML reports... 2022-03-17T21:14:59.6690755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211459.xml 2022-03-17T21:15:02.2044617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:02.2059770Z 2022-03-17T21:15:02.2060040Z Running tests... 2022-03-17T21:15:02.2060484Z ---------------------------------------------------------------------- 2022-03-17T21:15:02.2077395Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:02.2077682Z 2022-03-17T21:15:02.2077996Z ---------------------------------------------------------------------- 2022-03-17T21:15:02.2078596Z Ran 1 test in 0.002s 2022-03-17T21:15:02.2078725Z 2022-03-17T21:15:02.2078828Z OK (skipped=1) 2022-03-17T21:15:02.2078970Z 2022-03-17T21:15:02.2079077Z Generating XML reports... 2022-03-17T21:15:02.2117017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211502.xml 2022-03-17T21:15:04.6759610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:04.6770698Z 2022-03-17T21:15:04.6771025Z Running tests... 2022-03-17T21:15:04.6771616Z ---------------------------------------------------------------------- 2022-03-17T21:15:04.6790347Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:04.6790629Z 2022-03-17T21:15:04.6790939Z ---------------------------------------------------------------------- 2022-03-17T21:15:04.6791236Z Ran 1 test in 0.002s 2022-03-17T21:15:04.6791365Z 2022-03-17T21:15:04.6791502Z OK (skipped=1) 2022-03-17T21:15:04.6791665Z 2022-03-17T21:15:04.6791795Z Generating XML reports... 2022-03-17T21:15:04.6832819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211504.xml 2022-03-17T21:15:07.2073290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:07.2085367Z 2022-03-17T21:15:07.2085784Z Running tests... 2022-03-17T21:15:07.2086383Z ---------------------------------------------------------------------- 2022-03-17T21:15:07.2098223Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:07.2098539Z 2022-03-17T21:15:07.2098917Z ---------------------------------------------------------------------- 2022-03-17T21:15:07.2099271Z Ran 1 test in 0.002s 2022-03-17T21:15:07.2099443Z 2022-03-17T21:15:07.2099554Z OK (skipped=1) 2022-03-17T21:15:07.2099719Z 2022-03-17T21:15:07.2099845Z Generating XML reports... 2022-03-17T21:15:07.2139525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211507.xml 2022-03-17T21:15:09.9051512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:09.9061906Z 2022-03-17T21:15:09.9062337Z Running tests... 2022-03-17T21:15:09.9063297Z ---------------------------------------------------------------------- 2022-03-17T21:15:09.9085761Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:09.9086502Z 2022-03-17T21:15:09.9087297Z ---------------------------------------------------------------------- 2022-03-17T21:15:09.9088056Z Ran 1 test in 0.002s 2022-03-17T21:15:09.9088433Z 2022-03-17T21:15:09.9088661Z OK (skipped=1) 2022-03-17T21:15:09.9089008Z 2022-03-17T21:15:09.9089264Z Generating XML reports... 2022-03-17T21:15:09.9124214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211509.xml 2022-03-17T21:15:12.6025724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:12.6041042Z 2022-03-17T21:15:12.6041341Z Running tests... 2022-03-17T21:15:12.6041766Z ---------------------------------------------------------------------- 2022-03-17T21:15:12.6059649Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:12.6059921Z 2022-03-17T21:15:12.6060228Z ---------------------------------------------------------------------- 2022-03-17T21:15:12.6060513Z Ran 1 test in 0.002s 2022-03-17T21:15:12.6060653Z 2022-03-17T21:15:12.6060740Z OK (skipped=1) 2022-03-17T21:15:12.6060874Z 2022-03-17T21:15:12.6060969Z Generating XML reports... 2022-03-17T21:15:12.6100115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211512.xml 2022-03-17T21:15:15.0601760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:15.0613512Z 2022-03-17T21:15:15.0614013Z Running tests... 2022-03-17T21:15:15.0615186Z ---------------------------------------------------------------------- 2022-03-17T21:15:15.0629754Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:15:15.0630480Z 2022-03-17T21:15:15.0631282Z ---------------------------------------------------------------------- 2022-03-17T21:15:15.0632077Z Ran 1 test in 0.002s 2022-03-17T21:15:15.0632431Z 2022-03-17T21:15:15.0632669Z OK (skipped=1) 2022-03-17T21:15:15.0633024Z 2022-03-17T21:15:15.0633294Z Generating XML reports... 2022-03-17T21:15:15.0673370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211515.xml 2022-03-17T21:15:17.5274070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:17.5285758Z 2022-03-17T21:15:17.5286203Z Running tests... 2022-03-17T21:15:17.5286964Z ---------------------------------------------------------------------- 2022-03-17T21:15:17.5307603Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:17.5308128Z 2022-03-17T21:15:17.5308554Z ---------------------------------------------------------------------- 2022-03-17T21:15:17.5308918Z Ran 1 test in 0.002s 2022-03-17T21:15:17.5309074Z 2022-03-17T21:15:17.5309204Z OK (skipped=1) 2022-03-17T21:15:17.5309370Z 2022-03-17T21:15:17.5310348Z Generating XML reports... 2022-03-17T21:15:17.5349307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211517.xml 2022-03-17T21:15:20.0406036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:20.0419913Z 2022-03-17T21:15:20.0420174Z Running tests... 2022-03-17T21:15:20.0420605Z ---------------------------------------------------------------------- 2022-03-17T21:15:20.0438242Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:20.0438521Z 2022-03-17T21:15:20.0439036Z ---------------------------------------------------------------------- 2022-03-17T21:15:20.0439320Z Ran 1 test in 0.002s 2022-03-17T21:15:20.0439472Z 2022-03-17T21:15:20.0439559Z OK (skipped=1) 2022-03-17T21:15:20.0439692Z 2022-03-17T21:15:20.0439992Z Generating XML reports... 2022-03-17T21:15:20.0477745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211520.xml 2022-03-17T21:15:23.1238120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:23.1253087Z 2022-03-17T21:15:23.1253443Z Running tests... 2022-03-17T21:15:23.1254029Z ---------------------------------------------------------------------- 2022-03-17T21:15:23.1273197Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:23.1273538Z 2022-03-17T21:15:23.1273912Z ---------------------------------------------------------------------- 2022-03-17T21:15:23.1274274Z Ran 1 test in 0.002s 2022-03-17T21:15:23.1274447Z 2022-03-17T21:15:23.1274559Z OK (skipped=1) 2022-03-17T21:15:23.1274706Z 2022-03-17T21:15:23.1274838Z Generating XML reports... 2022-03-17T21:15:23.1313698Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211523.xml 2022-03-17T21:15:25.8633515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:25.8648778Z 2022-03-17T21:15:25.8649069Z Running tests... 2022-03-17T21:15:25.8649512Z ---------------------------------------------------------------------- 2022-03-17T21:15:25.8668930Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:25.8669377Z 2022-03-17T21:15:25.8669677Z ---------------------------------------------------------------------- 2022-03-17T21:15:25.8670016Z Ran 1 test in 0.002s 2022-03-17T21:15:25.8670187Z 2022-03-17T21:15:25.8670299Z OK (skipped=1) 2022-03-17T21:15:25.8670460Z 2022-03-17T21:15:25.8670574Z Generating XML reports... 2022-03-17T21:15:25.8710448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211525.xml 2022-03-17T21:15:28.5218488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:28.5231064Z 2022-03-17T21:15:28.5231443Z Running tests... 2022-03-17T21:15:28.5232905Z ---------------------------------------------------------------------- 2022-03-17T21:15:28.5250252Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:28.5251067Z 2022-03-17T21:15:28.5252032Z ---------------------------------------------------------------------- 2022-03-17T21:15:28.5253008Z Ran 1 test in 0.002s 2022-03-17T21:15:28.5253378Z 2022-03-17T21:15:28.5253589Z OK (skipped=1) 2022-03-17T21:15:28.5253944Z 2022-03-17T21:15:28.5254212Z Generating XML reports... 2022-03-17T21:15:28.5291216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211528.xml 2022-03-17T21:15:30.9980856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:30.9996467Z 2022-03-17T21:15:30.9996676Z Running tests... 2022-03-17T21:15:30.9997110Z ---------------------------------------------------------------------- 2022-03-17T21:15:31.0016021Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:31.0016304Z 2022-03-17T21:15:31.0016601Z ---------------------------------------------------------------------- 2022-03-17T21:15:31.0016887Z Ran 1 test in 0.002s 2022-03-17T21:15:31.0017028Z 2022-03-17T21:15:31.0017132Z OK (skipped=1) 2022-03-17T21:15:31.0017278Z 2022-03-17T21:15:31.0017415Z Generating XML reports... 2022-03-17T21:15:31.0056030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211530.xml 2022-03-17T21:15:33.9341014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:33.9352620Z 2022-03-17T21:15:33.9353249Z Running tests... 2022-03-17T21:15:33.9354855Z ---------------------------------------------------------------------- 2022-03-17T21:15:34.5367286Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39615 2022-03-17T21:15:34.5406652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39616 2022-03-17T21:15:34.5408247Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:15:35.7244357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:35.7245004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:35.7250060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:35.7250912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:35.7258810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:35.7261933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:36.6021849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:15:36.6197697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:15:36.6200299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:36.6226603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:36.6656689Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:15:36.7217424Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:15:38.4517208Z ok (4.516s) 2022-03-17T21:15:38.4517704Z 2022-03-17T21:15:38.4518933Z ---------------------------------------------------------------------- 2022-03-17T21:15:38.4519883Z Ran 1 test in 4.516s 2022-03-17T21:15:38.4520339Z 2022-03-17T21:15:38.4520579Z OK 2022-03-17T21:15:38.4520941Z 2022-03-17T21:15:38.4521352Z Generating XML reports... 2022-03-17T21:15:38.4590396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211533.xml 2022-03-17T21:15:41.3031154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:41.3045082Z 2022-03-17T21:15:41.3045801Z Running tests... 2022-03-17T21:15:41.3046992Z ---------------------------------------------------------------------- 2022-03-17T21:15:41.9231927Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39836 2022-03-17T21:15:41.9268693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39837 2022-03-17T21:15:41.9269998Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:15:43.0913615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:43.0914973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:43.0916912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:43.0918746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:43.1028538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:43.1937809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:44.9348413Z skip (3.630s) 2022-03-17T21:15:44.9348880Z 2022-03-17T21:15:44.9349890Z ---------------------------------------------------------------------- 2022-03-17T21:15:44.9350667Z Ran 1 test in 3.630s 2022-03-17T21:15:44.9351041Z 2022-03-17T21:15:44.9351277Z OK (skipped=1) 2022-03-17T21:15:44.9351607Z 2022-03-17T21:15:44.9351875Z Generating XML reports... 2022-03-17T21:15:44.9424527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211541.xml 2022-03-17T21:15:47.6616179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:47.6630224Z 2022-03-17T21:15:47.6630389Z Running tests... 2022-03-17T21:15:47.6631017Z ---------------------------------------------------------------------- 2022-03-17T21:15:47.6650571Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:47.6651198Z 2022-03-17T21:15:47.6652206Z ---------------------------------------------------------------------- 2022-03-17T21:15:47.6653010Z Ran 1 test in 0.002s 2022-03-17T21:15:47.6653378Z 2022-03-17T21:15:47.6653587Z OK (skipped=1) 2022-03-17T21:15:47.6653937Z 2022-03-17T21:15:47.6654205Z Generating XML reports... 2022-03-17T21:15:47.6690960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211547.xml 2022-03-17T21:15:50.2510016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:50.2522429Z 2022-03-17T21:15:50.2522737Z Running tests... 2022-03-17T21:15:50.2523282Z ---------------------------------------------------------------------- 2022-03-17T21:15:50.8677933Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40113 2022-03-17T21:15:50.8722336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40114 2022-03-17T21:15:50.8723808Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:15:52.0508921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:52.0510336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:52.0512279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:52.0514154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:52.0615641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:52.1520366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:54.6835500Z ok (4.430s) 2022-03-17T21:15:54.6835997Z 2022-03-17T21:15:54.6837138Z ---------------------------------------------------------------------- 2022-03-17T21:15:54.6837941Z Ran 1 test in 4.431s 2022-03-17T21:15:54.6838315Z 2022-03-17T21:15:54.6838505Z OK 2022-03-17T21:15:54.6838797Z 2022-03-17T21:15:54.6840546Z Generating XML reports... 2022-03-17T21:15:54.6910370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211550.xml 2022-03-17T21:15:57.6038941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:57.6053483Z 2022-03-17T21:15:57.6054010Z Running tests... 2022-03-17T21:15:57.6055219Z ---------------------------------------------------------------------- 2022-03-17T21:15:58.2172732Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40323 2022-03-17T21:15:58.2220447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40324 2022-03-17T21:15:58.2221850Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:15:59.3732400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:59.3732867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:59.3733575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:59.3734373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:59.3748638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:59.3749846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:02.3330736Z ok (4.727s) 2022-03-17T21:16:02.3331164Z 2022-03-17T21:16:02.3332029Z ---------------------------------------------------------------------- 2022-03-17T21:16:02.3332796Z Ran 1 test in 4.727s 2022-03-17T21:16:02.3333179Z 2022-03-17T21:16:02.3333350Z OK 2022-03-17T21:16:02.3333641Z 2022-03-17T21:16:02.3333912Z Generating XML reports... 2022-03-17T21:16:02.3413852Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211557.xml 2022-03-17T21:16:04.9633152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:04.9648405Z 2022-03-17T21:16:04.9648844Z Running tests... 2022-03-17T21:16:04.9649331Z ---------------------------------------------------------------------- 2022-03-17T21:16:05.5640939Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40533 2022-03-17T21:16:05.5682863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40534 2022-03-17T21:16:05.5684245Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:06.7250013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:06.7250484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:06.7251272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:06.7252086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:06.7366448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:06.8270481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:06.8381993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:16:06.8383054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:16:06.8384567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:06.8386086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:09.5798521Z ok (4.614s) 2022-03-17T21:16:09.5798938Z 2022-03-17T21:16:09.5799908Z ---------------------------------------------------------------------- 2022-03-17T21:16:09.5800689Z Ran 1 test in 4.615s 2022-03-17T21:16:09.5801060Z 2022-03-17T21:16:09.5801252Z OK 2022-03-17T21:16:09.5801535Z 2022-03-17T21:16:09.5801814Z Generating XML reports... 2022-03-17T21:16:09.5880754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211604.xml 2022-03-17T21:16:12.2101594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:12.2113359Z 2022-03-17T21:16:12.2113906Z Running tests... 2022-03-17T21:16:12.2115094Z ---------------------------------------------------------------------- 2022-03-17T21:16:12.8088171Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40749 2022-03-17T21:16:12.8127066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40750 2022-03-17T21:16:12.8128456Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:13.9747665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:13.9748853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:13.9749479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:13.9750236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:13.9856389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:14.0767210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:15.8227310Z skip (3.611s) 2022-03-17T21:16:15.8227821Z 2022-03-17T21:16:15.8228880Z ---------------------------------------------------------------------- 2022-03-17T21:16:15.8229650Z Ran 1 test in 3.611s 2022-03-17T21:16:15.8230014Z 2022-03-17T21:16:15.8230242Z OK (skipped=1) 2022-03-17T21:16:15.8230618Z 2022-03-17T21:16:15.8230872Z Generating XML reports... 2022-03-17T21:16:15.8306974Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211612.xml 2022-03-17T21:16:18.4571032Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:18.4584984Z 2022-03-17T21:16:18.4585231Z Running tests... 2022-03-17T21:16:18.4585679Z ---------------------------------------------------------------------- 2022-03-17T21:16:19.0585266Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40959 2022-03-17T21:16:19.0623420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40960 2022-03-17T21:16:19.0624762Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:20.1981288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:20.1982565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:20.1984440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:20.1986227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:20.2091409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:20.2997063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:21.9722432Z skip (3.513s) 2022-03-17T21:16:21.9722835Z 2022-03-17T21:16:21.9724785Z ---------------------------------------------------------------------- 2022-03-17T21:16:21.9725615Z Ran 1 test in 3.513s 2022-03-17T21:16:21.9725979Z 2022-03-17T21:16:21.9726246Z OK (skipped=1) 2022-03-17T21:16:21.9726596Z 2022-03-17T21:16:21.9726866Z Generating XML reports... 2022-03-17T21:16:21.9807522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211618.xml 2022-03-17T21:16:24.6625869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:24.6636540Z 2022-03-17T21:16:24.6636799Z Running tests... 2022-03-17T21:16:24.6638818Z ---------------------------------------------------------------------- 2022-03-17T21:16:25.2639091Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41169 2022-03-17T21:16:25.2682775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41170 2022-03-17T21:16:25.2684335Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:26.4341080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:26.4341703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:26.4342792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:26.4343625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:26.4458534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:26.5360495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:28.2783643Z skip (3.614s) 2022-03-17T21:16:28.2784247Z 2022-03-17T21:16:28.2785399Z ---------------------------------------------------------------------- 2022-03-17T21:16:28.2786220Z Ran 1 test in 3.614s 2022-03-17T21:16:28.2786611Z 2022-03-17T21:16:28.2786821Z OK (skipped=1) 2022-03-17T21:16:28.2787170Z 2022-03-17T21:16:28.2787448Z Generating XML reports... 2022-03-17T21:16:28.2863015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211624.xml 2022-03-17T21:16:30.9435405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:30.9448453Z 2022-03-17T21:16:30.9448786Z Running tests... 2022-03-17T21:16:30.9449939Z ---------------------------------------------------------------------- 2022-03-17T21:16:31.5803925Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41379 2022-03-17T21:16:31.5837955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41380 2022-03-17T21:16:31.5839255Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:32.7431183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:32.7432690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:32.7434769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:32.7533860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:32.7546603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:32.8447623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:32.8570226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:16:32.8571729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:16:32.8574244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:32.8576165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:35.7941539Z ok (4.849s) 2022-03-17T21:16:35.7942031Z 2022-03-17T21:16:35.7943122Z ---------------------------------------------------------------------- 2022-03-17T21:16:35.7944078Z Ran 1 test in 4.849s 2022-03-17T21:16:35.7944526Z 2022-03-17T21:16:35.7944785Z OK 2022-03-17T21:16:35.7945125Z 2022-03-17T21:16:35.7945450Z Generating XML reports... 2022-03-17T21:16:35.8023512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211630.xml 2022-03-17T21:16:38.4085517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:38.4098497Z 2022-03-17T21:16:38.4098650Z Running tests... 2022-03-17T21:16:38.4099174Z ---------------------------------------------------------------------- 2022-03-17T21:16:38.4119617Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:16:38.4120442Z 2022-03-17T21:16:38.4121430Z ---------------------------------------------------------------------- 2022-03-17T21:16:38.4122580Z Ran 1 test in 0.002s 2022-03-17T21:16:38.4123011Z 2022-03-17T21:16:38.4123233Z OK (skipped=1) 2022-03-17T21:16:38.4123584Z 2022-03-17T21:16:38.4123854Z Generating XML reports... 2022-03-17T21:16:38.4158767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211638.xml 2022-03-17T21:16:41.0496718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:41.0509436Z 2022-03-17T21:16:41.0510033Z Running tests... 2022-03-17T21:16:41.0511489Z ---------------------------------------------------------------------- 2022-03-17T21:16:41.6855974Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41662 2022-03-17T21:16:41.6898113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41663 2022-03-17T21:16:41.6899652Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:42.8388104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:42.8390781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:42.8393239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:42.8491423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:42.8494007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:42.9417845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:44.7982619Z skip (3.747s) 2022-03-17T21:16:44.7983068Z 2022-03-17T21:16:44.7984152Z ---------------------------------------------------------------------- 2022-03-17T21:16:44.7984979Z Ran 1 test in 3.748s 2022-03-17T21:16:44.7985353Z 2022-03-17T21:16:44.7985594Z OK (skipped=1) 2022-03-17T21:16:44.7985932Z 2022-03-17T21:16:44.7986209Z Generating XML reports... 2022-03-17T21:16:44.8065227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211641.xml 2022-03-17T21:16:47.4659176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:47.4670534Z 2022-03-17T21:16:47.4670744Z Running tests... 2022-03-17T21:16:47.4671287Z ---------------------------------------------------------------------- 2022-03-17T21:16:48.1041607Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41872 2022-03-17T21:16:48.1082431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41873 2022-03-17T21:16:48.1083755Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:49.2265391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:49.2267083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:49.2268131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:49.2269471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:49.2373108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:49.3279754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:51.1179321Z ok (3.650s) 2022-03-17T21:16:51.1179900Z 2022-03-17T21:16:51.1180954Z ---------------------------------------------------------------------- 2022-03-17T21:16:51.1181896Z Ran 1 test in 3.650s 2022-03-17T21:16:51.1182336Z 2022-03-17T21:16:51.1182575Z OK 2022-03-17T21:16:51.1183066Z 2022-03-17T21:16:51.1183511Z Generating XML reports... 2022-03-17T21:16:51.1260474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211647.xml 2022-03-17T21:16:53.7708039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:53.7720846Z 2022-03-17T21:16:53.7721192Z Running tests... 2022-03-17T21:16:53.7721720Z ---------------------------------------------------------------------- 2022-03-17T21:16:54.3996065Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42082 2022-03-17T21:16:54.4031909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42083 2022-03-17T21:16:54.4033244Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:16:55.5394547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:55.5395459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:55.5396039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:55.5396802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:55.5409733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:55.6406839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:57.3127441Z ok (3.540s) 2022-03-17T21:16:57.3127837Z 2022-03-17T21:16:57.3128775Z ---------------------------------------------------------------------- 2022-03-17T21:16:57.3129553Z Ran 1 test in 3.540s 2022-03-17T21:16:57.3129915Z 2022-03-17T21:16:57.3130099Z OK 2022-03-17T21:16:57.3135273Z 2022-03-17T21:16:57.3135806Z Generating XML reports... 2022-03-17T21:16:57.3208818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211653.xml 2022-03-17T21:16:59.8674660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:59.8685485Z 2022-03-17T21:16:59.8685803Z Running tests... 2022-03-17T21:16:59.8686383Z ---------------------------------------------------------------------- 2022-03-17T21:16:59.8709286Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:16:59.8709628Z 2022-03-17T21:16:59.8710068Z ---------------------------------------------------------------------- 2022-03-17T21:16:59.8710430Z Ran 1 test in 0.002s 2022-03-17T21:16:59.8710590Z 2022-03-17T21:16:59.8710696Z OK (skipped=1) 2022-03-17T21:16:59.8710858Z 2022-03-17T21:16:59.8710984Z Generating XML reports... 2022-03-17T21:16:59.8747064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211659.xml 2022-03-17T21:17:02.3634127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:02.3647602Z 2022-03-17T21:17:02.3647939Z Running tests... 2022-03-17T21:17:02.3648496Z ---------------------------------------------------------------------- 2022-03-17T21:17:02.3673326Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:17:02.3673624Z 2022-03-17T21:17:02.3673999Z ---------------------------------------------------------------------- 2022-03-17T21:17:02.3674361Z Ran 1 test in 0.003s 2022-03-17T21:17:02.3674532Z 2022-03-17T21:17:02.3674630Z OK (skipped=1) 2022-03-17T21:17:02.3674792Z 2022-03-17T21:17:02.3674932Z Generating XML reports... 2022-03-17T21:17:02.3712505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211702.xml 2022-03-17T21:17:04.9164746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:04.9180395Z 2022-03-17T21:17:04.9180730Z Running tests... 2022-03-17T21:17:04.9181283Z ---------------------------------------------------------------------- 2022-03-17T21:17:04.9205200Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:17:04.9205582Z 2022-03-17T21:17:04.9206069Z ---------------------------------------------------------------------- 2022-03-17T21:17:04.9206525Z Ran 1 test in 0.003s 2022-03-17T21:17:04.9206734Z 2022-03-17T21:17:04.9206851Z OK (skipped=1) 2022-03-17T21:17:04.9207074Z 2022-03-17T21:17:04.9207205Z Generating XML reports... 2022-03-17T21:17:04.9248693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211704.xml 2022-03-17T21:17:07.4492424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:07.4502767Z 2022-03-17T21:17:07.4503117Z Running tests... 2022-03-17T21:17:07.4503672Z ---------------------------------------------------------------------- 2022-03-17T21:17:07.4524893Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:17:07.4525189Z 2022-03-17T21:17:07.4525561Z ---------------------------------------------------------------------- 2022-03-17T21:17:07.4525922Z Ran 1 test in 0.002s 2022-03-17T21:17:07.4526091Z 2022-03-17T21:17:07.4526213Z OK (skipped=1) 2022-03-17T21:17:07.4526370Z 2022-03-17T21:17:07.4526496Z Generating XML reports... 2022-03-17T21:17:07.4562948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211707.xml 2022-03-17T21:17:09.9979822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:09.9994738Z 2022-03-17T21:17:09.9995166Z Running tests... 2022-03-17T21:17:09.9996350Z ---------------------------------------------------------------------- 2022-03-17T21:17:10.0015443Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:17:10.0016310Z 2022-03-17T21:17:10.0017279Z ---------------------------------------------------------------------- 2022-03-17T21:17:10.0018245Z Ran 1 test in 0.002s 2022-03-17T21:17:10.0018616Z 2022-03-17T21:17:10.0018855Z OK (skipped=1) 2022-03-17T21:17:10.0019210Z 2022-03-17T21:17:10.0019478Z Generating XML reports... 2022-03-17T21:17:10.0061065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211709.xml 2022-03-17T21:17:12.7223292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:12.7234255Z 2022-03-17T21:17:12.7234936Z Running tests... 2022-03-17T21:17:12.7236444Z ---------------------------------------------------------------------- 2022-03-17T21:17:12.7259804Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:17:12.7260649Z 2022-03-17T21:17:12.7261637Z ---------------------------------------------------------------------- 2022-03-17T21:17:12.7262608Z Ran 1 test in 0.002s 2022-03-17T21:17:12.7262988Z 2022-03-17T21:17:12.7263220Z OK (skipped=1) 2022-03-17T21:17:12.7263575Z 2022-03-17T21:17:12.7263852Z Generating XML reports... 2022-03-17T21:17:12.7301264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211712.xml 2022-03-17T21:17:15.2431677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:15.2446525Z 2022-03-17T21:17:15.2446944Z Running tests... 2022-03-17T21:17:15.2447479Z ---------------------------------------------------------------------- 2022-03-17T21:17:15.2469331Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:17:15.2469656Z 2022-03-17T21:17:15.2470025Z ---------------------------------------------------------------------- 2022-03-17T21:17:15.2470579Z Ran 1 test in 0.002s 2022-03-17T21:17:15.2470747Z 2022-03-17T21:17:15.2470860Z OK (skipped=1) 2022-03-17T21:17:15.2471034Z 2022-03-17T21:17:15.2471160Z Generating XML reports... 2022-03-17T21:17:15.2512695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211715.xml 2022-03-17T21:17:17.9288410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:17.9299574Z 2022-03-17T21:17:17.9300246Z Running tests... 2022-03-17T21:17:17.9301693Z ---------------------------------------------------------------------- 2022-03-17T21:17:18.5299843Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42761 2022-03-17T21:17:18.5332374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42762 2022-03-17T21:17:18.5334001Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:19.6944667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:19.6947121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:19.6948425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:19.6950094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:19.7053972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:19.7963464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:21.6434665Z ok (3.713s) 2022-03-17T21:17:21.6435073Z 2022-03-17T21:17:21.6436123Z ---------------------------------------------------------------------- 2022-03-17T21:17:21.6436936Z Ran 1 test in 3.714s 2022-03-17T21:17:21.6437322Z 2022-03-17T21:17:21.6437493Z OK 2022-03-17T21:17:21.6437785Z 2022-03-17T21:17:21.6438073Z Generating XML reports... 2022-03-17T21:17:21.6508338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211717.xml 2022-03-17T21:17:24.2078318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:24.2091465Z 2022-03-17T21:17:24.2091803Z Running tests... 2022-03-17T21:17:24.2092388Z ---------------------------------------------------------------------- 2022-03-17T21:17:24.8390008Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42977 2022-03-17T21:17:24.8425461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42978 2022-03-17T21:17:24.8427093Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:25.9664324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:25.9664915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:25.9665816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:25.9666619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:25.9775002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:26.0680115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:29.0536066Z ok (4.844s) 2022-03-17T21:17:29.0536433Z 2022-03-17T21:17:29.0537919Z ---------------------------------------------------------------------- 2022-03-17T21:17:29.0538691Z Ran 1 test in 4.844s 2022-03-17T21:17:29.0539077Z 2022-03-17T21:17:29.0539245Z OK 2022-03-17T21:17:29.0539538Z 2022-03-17T21:17:29.0539826Z Generating XML reports... 2022-03-17T21:17:29.0617165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211724.xml 2022-03-17T21:17:31.7734700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:31.7750789Z 2022-03-17T21:17:31.7750926Z Running tests... 2022-03-17T21:17:31.7751577Z ---------------------------------------------------------------------- 2022-03-17T21:17:32.4097965Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43193 2022-03-17T21:17:32.4149669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43194 2022-03-17T21:17:32.4151031Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:33.5567927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:33.5569149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:33.5571154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:33.5572918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:33.5581721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:33.5582891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:33.5695138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:17:33.5696392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:17:33.5698183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:33.5795892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:35.5230684Z ok (3.747s) 2022-03-17T21:17:35.5231217Z 2022-03-17T21:17:35.5232331Z ---------------------------------------------------------------------- 2022-03-17T21:17:35.5233139Z Ran 1 test in 3.748s 2022-03-17T21:17:35.5233517Z 2022-03-17T21:17:35.5233684Z OK 2022-03-17T21:17:35.5233980Z 2022-03-17T21:17:35.5234268Z Generating XML reports... 2022-03-17T21:17:35.5304450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211731.xml 2022-03-17T21:17:38.1700564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:38.1714491Z 2022-03-17T21:17:38.1715166Z Running tests... 2022-03-17T21:17:38.1715705Z ---------------------------------------------------------------------- 2022-03-17T21:17:38.7704543Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43415 2022-03-17T21:17:38.7743634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43416 2022-03-17T21:17:38.7744963Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:39.9236109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:39.9238161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:39.9240260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:39.9242766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:39.9247062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:39.9248217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:41.6826405Z skip (3.511s) 2022-03-17T21:17:41.6826852Z 2022-03-17T21:17:41.6827896Z ---------------------------------------------------------------------- 2022-03-17T21:17:41.6828773Z Ran 1 test in 3.511s 2022-03-17T21:17:41.6829164Z 2022-03-17T21:17:41.6829412Z OK (skipped=1) 2022-03-17T21:17:41.6829769Z 2022-03-17T21:17:41.6830055Z Generating XML reports... 2022-03-17T21:17:41.6901917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211738.xml 2022-03-17T21:17:44.3867858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:44.3878967Z 2022-03-17T21:17:44.3879119Z Running tests... 2022-03-17T21:17:44.3879814Z ---------------------------------------------------------------------- 2022-03-17T21:17:45.0204586Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43625 2022-03-17T21:17:45.0243718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43626 2022-03-17T21:17:45.0245042Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:46.1520554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:46.1521787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:46.1524336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:46.1526191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:46.1527477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:46.1529057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:48.2345742Z ok (3.846s) 2022-03-17T21:17:48.2346128Z 2022-03-17T21:17:48.2347173Z ---------------------------------------------------------------------- 2022-03-17T21:17:48.2347938Z Ran 1 test in 3.846s 2022-03-17T21:17:48.2348297Z 2022-03-17T21:17:48.2348481Z OK 2022-03-17T21:17:48.2348782Z 2022-03-17T21:17:48.2349056Z Generating XML reports... 2022-03-17T21:17:48.2423815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211744.xml 2022-03-17T21:17:50.8341424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:50.8350714Z 2022-03-17T21:17:50.8350877Z Running tests... 2022-03-17T21:17:50.8351414Z ---------------------------------------------------------------------- 2022-03-17T21:17:51.4349695Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43835 2022-03-17T21:17:51.4396291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43836 2022-03-17T21:17:51.4397728Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:52.5801414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:52.5802941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:52.5805412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:52.5903186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:52.5904542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:52.6820139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:54.5529466Z ok (3.717s) 2022-03-17T21:17:54.5529892Z 2022-03-17T21:17:54.5530952Z ---------------------------------------------------------------------- 2022-03-17T21:17:54.5531730Z Ran 1 test in 3.717s 2022-03-17T21:17:54.5532096Z 2022-03-17T21:17:54.5532303Z OK 2022-03-17T21:17:54.5532581Z 2022-03-17T21:17:54.5532870Z Generating XML reports... 2022-03-17T21:17:54.5611365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211750.xml 2022-03-17T21:17:57.2211313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:57.2230266Z 2022-03-17T21:17:57.2230624Z Running tests... 2022-03-17T21:17:57.2231180Z ---------------------------------------------------------------------- 2022-03-17T21:17:57.8490260Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44045 2022-03-17T21:17:57.8537762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44046 2022-03-17T21:17:57.8539077Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:17:58.9875107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:58.9878629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:58.9879693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:58.9981763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:58.9995687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:59.0895017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:00.8625912Z skip (3.639s) 2022-03-17T21:18:00.8626301Z 2022-03-17T21:18:00.8627882Z ---------------------------------------------------------------------- 2022-03-17T21:18:00.8628655Z Ran 1 test in 3.640s 2022-03-17T21:18:00.8629044Z 2022-03-17T21:18:00.8629276Z OK (skipped=1) 2022-03-17T21:18:00.8629627Z 2022-03-17T21:18:00.8629897Z Generating XML reports... 2022-03-17T21:18:00.8705808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211757.xml 2022-03-17T21:18:03.4397635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:03.4409191Z 2022-03-17T21:18:03.4409888Z Running tests... 2022-03-17T21:18:03.4411061Z ---------------------------------------------------------------------- 2022-03-17T21:18:04.0395136Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44255 2022-03-17T21:18:04.0450166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44256 2022-03-17T21:18:04.0451858Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:18:05.1819102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:05.1840012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:05.1841756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:05.1920877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:05.1924362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:05.2857603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:06.9529876Z skip (3.512s) 2022-03-17T21:18:06.9530273Z 2022-03-17T21:18:06.9531133Z ---------------------------------------------------------------------- 2022-03-17T21:18:06.9531914Z Ran 1 test in 3.512s 2022-03-17T21:18:06.9532278Z 2022-03-17T21:18:06.9532514Z OK (skipped=1) 2022-03-17T21:18:06.9532856Z 2022-03-17T21:18:06.9533143Z Generating XML reports... 2022-03-17T21:18:06.9613509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211803.xml 2022-03-17T21:18:09.6095550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:09.6106686Z 2022-03-17T21:18:09.6107156Z Running tests... 2022-03-17T21:18:09.6108163Z ---------------------------------------------------------------------- 2022-03-17T21:18:10.2319906Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44465 2022-03-17T21:18:10.2363993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44466 2022-03-17T21:18:10.2365343Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:18:11.3787901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:11.3789156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:11.3791017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:11.3890864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:11.3893475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:11.4807827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:21.0475780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:21.0483268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:23.7644237Z ok (14.153s) 2022-03-17T21:18:23.7644756Z 2022-03-17T21:18:23.7646104Z ---------------------------------------------------------------------- 2022-03-17T21:18:23.7647153Z Ran 1 test in 14.154s 2022-03-17T21:18:23.7647588Z 2022-03-17T21:18:23.7647835Z OK 2022-03-17T21:18:23.7648234Z 2022-03-17T21:18:23.7648530Z Generating XML reports... 2022-03-17T21:18:23.7729322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211809.xml 2022-03-17T21:18:26.3988301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:26.3997872Z 2022-03-17T21:18:26.3998396Z Running tests... 2022-03-17T21:18:26.3999533Z ---------------------------------------------------------------------- 2022-03-17T21:18:27.0003891Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44679 2022-03-17T21:18:27.0039560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44680 2022-03-17T21:18:27.0041314Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:18:28.1664561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:28.1665676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:28.1666293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:28.1667049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:28.1781203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:28.2685891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:37.9085516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:37.9087765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:37.9113418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:37.9114631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:40.6295653Z ok (14.229s) 2022-03-17T21:18:40.6296245Z 2022-03-17T21:18:40.6297470Z ---------------------------------------------------------------------- 2022-03-17T21:18:40.6299845Z Ran 1 test in 14.230s 2022-03-17T21:18:40.6300367Z 2022-03-17T21:18:40.6300592Z OK 2022-03-17T21:18:40.6300974Z 2022-03-17T21:18:40.6301344Z Generating XML reports... 2022-03-17T21:18:40.6377261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211826.xml 2022-03-17T21:18:43.1961189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:43.1970276Z 2022-03-17T21:18:43.1970800Z Running tests... 2022-03-17T21:18:43.1972340Z ---------------------------------------------------------------------- 2022-03-17T21:18:43.7964339Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44893 2022-03-17T21:18:43.8003586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44894 2022-03-17T21:18:43.8005330Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:18:44.9374342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:44.9376644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:44.9377959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:44.9379648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:44.9390792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:45.0387923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:54.6291965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6295235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6324466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6325991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6588389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6590197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6607599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:54.6608809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:18:57.3308286Z ok (14.133s) 2022-03-17T21:18:57.3308914Z 2022-03-17T21:18:57.3310427Z ---------------------------------------------------------------------- 2022-03-17T21:18:57.3311287Z Ran 1 test in 14.133s 2022-03-17T21:18:57.3311801Z 2022-03-17T21:18:57.3312069Z OK 2022-03-17T21:18:57.3312467Z 2022-03-17T21:18:57.3312793Z Generating XML reports... 2022-03-17T21:18:57.3387483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211843.xml 2022-03-17T21:18:59.9481334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:59.9493530Z 2022-03-17T21:18:59.9494143Z Running tests... 2022-03-17T21:18:59.9495615Z ---------------------------------------------------------------------- 2022-03-17T21:19:00.5918496Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45107 2022-03-17T21:19:00.5953451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45108 2022-03-17T21:19:00.5955010Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:19:01.7532743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:01.7535126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:01.7536433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:01.7538140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:01.7642920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:01.8554366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:11.4284839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.4286356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.4317593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.4319046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.5242989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.5244495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.5252873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:11.5254303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:19:14.2289402Z ok (14.279s) 2022-03-17T21:19:14.2289981Z 2022-03-17T21:19:14.2291363Z ---------------------------------------------------------------------- 2022-03-17T21:19:14.2292247Z Ran 1 test in 14.279s 2022-03-17T21:19:14.2292775Z 2022-03-17T21:19:14.2293043Z OK 2022-03-17T21:19:14.2294165Z 2022-03-17T21:19:14.2294840Z Generating XML reports... 2022-03-17T21:19:14.2373507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211859.xml 2022-03-17T21:19:16.8368257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:16.8382574Z 2022-03-17T21:19:16.8383723Z Running tests... 2022-03-17T21:19:16.8384900Z ---------------------------------------------------------------------- 2022-03-17T21:19:17.4808854Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45321 2022-03-17T21:19:17.4848564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45322 2022-03-17T21:19:17.4849990Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:19:18.6200646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:18.6202142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:18.6204209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:18.6302357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:18.6304117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:18.7222972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:19.1291099Z 2022-03-17T21:19:21.3953065Z ok (4.557s) 2022-03-17T21:19:21.3953406Z 2022-03-17T21:19:21.3954296Z ---------------------------------------------------------------------- 2022-03-17T21:19:21.3955047Z Ran 1 test in 4.557s 2022-03-17T21:19:21.3955414Z 2022-03-17T21:19:21.3955609Z OK 2022-03-17T21:19:21.3955898Z 2022-03-17T21:19:21.3956172Z Generating XML reports... 2022-03-17T21:19:21.4026580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211916.xml 2022-03-17T21:19:24.0325014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:24.0337119Z 2022-03-17T21:19:24.0337617Z Running tests... 2022-03-17T21:19:24.0338779Z ---------------------------------------------------------------------- 2022-03-17T21:19:24.6695846Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45531 2022-03-17T21:19:24.6755801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45532 2022-03-17T21:19:24.6757127Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:19:25.8474976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:25.8476868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:25.8478180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:25.8479892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:25.8490523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:25.9489837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:27.8852263Z ok (3.851s) 2022-03-17T21:19:27.8852655Z 2022-03-17T21:19:27.8853578Z ---------------------------------------------------------------------- 2022-03-17T21:19:27.8854372Z Ran 1 test in 3.851s 2022-03-17T21:19:27.8854737Z 2022-03-17T21:19:27.8854921Z OK 2022-03-17T21:19:27.8855194Z 2022-03-17T21:19:27.8855468Z Generating XML reports... 2022-03-17T21:19:27.8937313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211924.xml 2022-03-17T21:19:30.4786259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:30.4798391Z 2022-03-17T21:19:30.4799685Z Running tests... 2022-03-17T21:19:30.4800854Z ---------------------------------------------------------------------- 2022-03-17T21:19:31.0900385Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45741 2022-03-17T21:19:31.0934777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45742 2022-03-17T21:19:31.0936460Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:19:32.2433613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:32.2452100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:32.2453011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:32.2534610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:32.2536980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:32.3464975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:34.1011169Z skip (3.621s) 2022-03-17T21:19:34.1012374Z 2022-03-17T21:19:34.1013793Z ---------------------------------------------------------------------- 2022-03-17T21:19:34.1014711Z Ran 1 test in 3.621s 2022-03-17T21:19:34.1015086Z 2022-03-17T21:19:34.1015362Z OK (skipped=1) 2022-03-17T21:19:34.1015695Z 2022-03-17T21:19:34.1015983Z Generating XML reports... 2022-03-17T21:19:34.1094607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211930.xml 2022-03-17T21:19:36.8152338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:36.8159323Z 2022-03-17T21:19:36.8159446Z Running tests... 2022-03-17T21:19:36.8160298Z ---------------------------------------------------------------------- 2022-03-17T21:19:37.4248250Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45951 2022-03-17T21:19:37.4295554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45952 2022-03-17T21:19:37.4298854Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:19:38.5787588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:38.5788804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:38.5790721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:38.5792483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:38.5800033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:38.5803573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:48.2429971Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:19:51.1606709Z ok (14.344s) 2022-03-17T21:19:51.1607128Z 2022-03-17T21:19:51.1608199Z ---------------------------------------------------------------------- 2022-03-17T21:19:51.1608988Z Ran 1 test in 14.344s 2022-03-17T21:19:51.1609351Z 2022-03-17T21:19:51.1609538Z OK 2022-03-17T21:19:51.1609835Z 2022-03-17T21:19:51.1610099Z Generating XML reports... 2022-03-17T21:19:51.1688973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211936.xml 2022-03-17T21:19:53.8154977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:53.8167485Z 2022-03-17T21:19:53.8167807Z Running tests... 2022-03-17T21:19:53.8168355Z ---------------------------------------------------------------------- 2022-03-17T21:19:54.4479665Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46165 2022-03-17T21:19:54.4527435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46166 2022-03-17T21:19:54.4529333Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:19:55.6458639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:55.6459954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:55.6461984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:55.6463769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:55.6574647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:55.7478141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:05.5468513Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:20:05.5483264Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:20:08.5821389Z ok (14.765s) 2022-03-17T21:20:08.5821767Z 2022-03-17T21:20:08.5822772Z ---------------------------------------------------------------------- 2022-03-17T21:20:08.5825946Z Ran 1 test in 14.765s 2022-03-17T21:20:08.5826393Z 2022-03-17T21:20:08.5826589Z OK 2022-03-17T21:20:08.5826969Z 2022-03-17T21:20:08.5827244Z Generating XML reports... 2022-03-17T21:20:08.5904292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211953.xml 2022-03-17T21:20:11.1936321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:11.1946583Z 2022-03-17T21:20:11.1947787Z Running tests... 2022-03-17T21:20:11.1948610Z ---------------------------------------------------------------------- 2022-03-17T21:20:11.7996414Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46379 2022-03-17T21:20:11.8040248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46380 2022-03-17T21:20:11.8041807Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:20:12.9806326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:12.9807853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:12.9809924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:12.9908689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:12.9925834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:13.0918431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:13.0953731Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.0957809Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.0967000Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:989.) 2022-03-17T21:20:13.0970152Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T21:20:13.0974820Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:989.) 2022-03-17T21:20:13.0977502Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T21:20:13.0978668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:20:13.0979867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:20:13.0983502Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.0987692Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.0992676Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.0996767Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.1001312Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.1006322Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.1009345Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.1012420Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.1015364Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:13.1018326Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:20:14.9130212Z ok (3.718s) 2022-03-17T21:20:14.9130798Z 2022-03-17T21:20:14.9132308Z ---------------------------------------------------------------------- 2022-03-17T21:20:14.9133362Z Ran 1 test in 3.718s 2022-03-17T21:20:14.9133894Z 2022-03-17T21:20:14.9134140Z OK 2022-03-17T21:20:14.9134506Z 2022-03-17T21:20:14.9134867Z Generating XML reports... 2022-03-17T21:20:14.9214746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212011.xml 2022-03-17T21:20:17.6697608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:17.6709376Z 2022-03-17T21:20:17.6709844Z Running tests... 2022-03-17T21:20:17.6711073Z ---------------------------------------------------------------------- 2022-03-17T21:20:18.2800000Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46593 2022-03-17T21:20:18.2841365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46594 2022-03-17T21:20:18.2843077Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:20:19.4210820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:19.4212064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:19.4213903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:19.4215695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:19.4322226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:19.5234407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:29.0155406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:20:29.0160994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:20:31.7167032Z ok (14.045s) 2022-03-17T21:20:31.7167456Z 2022-03-17T21:20:31.7168573Z ---------------------------------------------------------------------- 2022-03-17T21:20:31.7169355Z Ran 1 test in 14.045s 2022-03-17T21:20:31.7169731Z 2022-03-17T21:20:31.7169931Z OK 2022-03-17T21:20:31.7170203Z 2022-03-17T21:20:31.7170536Z Generating XML reports... 2022-03-17T21:20:31.7250930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212017.xml 2022-03-17T21:20:34.3119672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:34.3132065Z 2022-03-17T21:20:34.3132816Z Running tests... 2022-03-17T21:20:34.3134230Z ---------------------------------------------------------------------- 2022-03-17T21:20:34.9222606Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46807 2022-03-17T21:20:34.9264278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46808 2022-03-17T21:20:34.9265819Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:20:36.0860214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:36.0861691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:36.0863923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:36.0961247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:36.0971564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:36.1970854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:36.6101603Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:20:36.6104526Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:20:36.6124522Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:20:36.6126980Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:20:48.8572123Z ok (14.543s) 2022-03-17T21:20:48.8572536Z 2022-03-17T21:20:48.8573807Z ---------------------------------------------------------------------- 2022-03-17T21:20:48.8574626Z Ran 1 test in 14.544s 2022-03-17T21:20:48.8575019Z 2022-03-17T21:20:48.8575237Z OK 2022-03-17T21:20:48.8575548Z 2022-03-17T21:20:48.8575852Z Generating XML reports... 2022-03-17T21:20:48.8651657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212034.xml 2022-03-17T21:20:51.7398521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:51.7408808Z 2022-03-17T21:20:51.7409320Z Running tests... 2022-03-17T21:20:51.7410650Z ---------------------------------------------------------------------- 2022-03-17T21:20:52.3797739Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47083 2022-03-17T21:20:52.3835861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47084 2022-03-17T21:20:52.3837520Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:20:53.5302006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:53.5304341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:53.5305664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:53.5307307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:53.5412612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:53.6322512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:05.5054285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:05.5055526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:10.7158414Z ok (18.974s) 2022-03-17T21:21:10.7159445Z 2022-03-17T21:21:10.7160590Z ---------------------------------------------------------------------- 2022-03-17T21:21:10.7161369Z Ran 1 test in 18.975s 2022-03-17T21:21:10.7161765Z 2022-03-17T21:21:10.7162187Z OK 2022-03-17T21:21:10.7162491Z 2022-03-17T21:21:10.7162758Z Generating XML reports... 2022-03-17T21:21:10.7244542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212051.xml 2022-03-17T21:21:13.5324188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:13.5333305Z 2022-03-17T21:21:13.5333703Z Running tests... 2022-03-17T21:21:13.5334633Z ---------------------------------------------------------------------- 2022-03-17T21:21:14.1400654Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47297 2022-03-17T21:21:14.1434685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47298 2022-03-17T21:21:14.1436514Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:21:15.3183230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:15.3184256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:15.3184866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:15.3185619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:15.3293850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:15.4203777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:24.9509392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.9510687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:27.5753396Z ok (14.041s) 2022-03-17T21:21:27.5753798Z 2022-03-17T21:21:27.5754835Z ---------------------------------------------------------------------- 2022-03-17T21:21:27.5755595Z Ran 1 test in 14.042s 2022-03-17T21:21:27.5755962Z 2022-03-17T21:21:27.5760349Z OK 2022-03-17T21:21:27.5760728Z 2022-03-17T21:21:27.5761050Z Generating XML reports... 2022-03-17T21:21:27.5819260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212113.xml 2022-03-17T21:21:30.3808294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:30.3820351Z 2022-03-17T21:21:30.3820563Z Running tests... 2022-03-17T21:21:30.3821008Z ---------------------------------------------------------------------- 2022-03-17T21:21:30.9858084Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47511 2022-03-17T21:21:30.9941622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47512 2022-03-17T21:21:30.9942955Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:21:32.0909561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:32.0910822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:32.0912683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:32.1012627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:32.1021934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:32.1930650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:34.5081213Z skip (4.126s) 2022-03-17T21:21:34.5081763Z 2022-03-17T21:21:34.5082941Z ---------------------------------------------------------------------- 2022-03-17T21:21:34.5083707Z Ran 1 test in 4.126s 2022-03-17T21:21:34.5084058Z 2022-03-17T21:21:34.5084282Z OK (skipped=1) 2022-03-17T21:21:34.5084644Z 2022-03-17T21:21:34.5085131Z Generating XML reports... 2022-03-17T21:21:34.5164451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212130.xml 2022-03-17T21:21:37.1517268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:37.1529797Z 2022-03-17T21:21:37.1529948Z Running tests... 2022-03-17T21:21:37.1530511Z ---------------------------------------------------------------------- 2022-03-17T21:21:37.7594630Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47721 2022-03-17T21:21:37.7638065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47722 2022-03-17T21:21:37.7641519Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:21:38.8891992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:38.8892928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:38.8893532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:38.8894291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:38.8896792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:38.8900085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:40.5704392Z skip (3.417s) 2022-03-17T21:21:40.5704817Z 2022-03-17T21:21:40.5706088Z ---------------------------------------------------------------------- 2022-03-17T21:21:40.5706900Z Ran 1 test in 3.417s 2022-03-17T21:21:40.5707274Z 2022-03-17T21:21:40.5707542Z OK (skipped=1) 2022-03-17T21:21:40.5707910Z 2022-03-17T21:21:40.5708182Z Generating XML reports... 2022-03-17T21:21:40.5783011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212137.xml 2022-03-17T21:21:43.1996452Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:43.2008627Z 2022-03-17T21:21:43.2009174Z Running tests... 2022-03-17T21:21:43.2010372Z ---------------------------------------------------------------------- 2022-03-17T21:21:43.8024193Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47931 2022-03-17T21:21:43.8059874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47932 2022-03-17T21:21:43.8061190Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:21:44.9716895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:44.9717463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:44.9718322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:44.9719130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:44.9830447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:45.0735714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:46.7160230Z skip (3.514s) 2022-03-17T21:21:46.7160645Z 2022-03-17T21:21:46.7161499Z ---------------------------------------------------------------------- 2022-03-17T21:21:46.7162493Z Ran 1 test in 3.515s 2022-03-17T21:21:46.7162893Z 2022-03-17T21:21:46.7163124Z OK (skipped=1) 2022-03-17T21:21:46.7163470Z 2022-03-17T21:21:46.7163733Z Generating XML reports... 2022-03-17T21:21:46.7243973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212143.xml 2022-03-17T21:21:49.3869490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:49.3880040Z 2022-03-17T21:21:49.3880321Z Running tests... 2022-03-17T21:21:49.3882595Z ---------------------------------------------------------------------- 2022-03-17T21:21:50.0025712Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48141 2022-03-17T21:21:50.0091963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48142 2022-03-17T21:21:50.0092957Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:21:51.1761731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:51.1763264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:51.1765329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:51.1767150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:51.1870228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:51.2780027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:53.0171687Z skip (3.629s) 2022-03-17T21:21:53.0172085Z 2022-03-17T21:21:53.0173032Z ---------------------------------------------------------------------- 2022-03-17T21:21:53.0173818Z Ran 1 test in 3.629s 2022-03-17T21:21:53.0174181Z 2022-03-17T21:21:53.0174404Z OK (skipped=1) 2022-03-17T21:21:53.0174747Z 2022-03-17T21:21:53.0175012Z Generating XML reports... 2022-03-17T21:21:53.0258097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212149.xml 2022-03-17T21:21:55.8974229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:55.8985861Z 2022-03-17T21:21:55.8986404Z Running tests... 2022-03-17T21:21:55.8987044Z ---------------------------------------------------------------------- 2022-03-17T21:21:56.5036461Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48351 2022-03-17T21:21:56.5073727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48352 2022-03-17T21:21:56.5075605Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:21:57.6808995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:57.6810027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:57.6811562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:57.6813076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:57.6824861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:57.6825850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:00.2160013Z skip (4.317s) 2022-03-17T21:22:00.2160335Z 2022-03-17T21:22:00.2162312Z ---------------------------------------------------------------------- 2022-03-17T21:22:00.2163332Z Ran 1 test in 4.317s 2022-03-17T21:22:00.2163718Z 2022-03-17T21:22:00.2163983Z OK (skipped=1) 2022-03-17T21:22:00.2164333Z 2022-03-17T21:22:00.2164632Z Generating XML reports... 2022-03-17T21:22:00.2243364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212155.xml 2022-03-17T21:22:02.7935219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:02.7946500Z 2022-03-17T21:22:02.7947097Z Running tests... 2022-03-17T21:22:02.7948315Z ---------------------------------------------------------------------- 2022-03-17T21:22:03.3975229Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48561 2022-03-17T21:22:03.4015130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48562 2022-03-17T21:22:03.4016605Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:04.5225192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:04.5227141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:04.5228490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:04.5230231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:04.5330749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:04.6239677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:06.4108610Z skip (3.616s) 2022-03-17T21:22:06.4109038Z 2022-03-17T21:22:06.4109913Z ---------------------------------------------------------------------- 2022-03-17T21:22:06.4110668Z Ran 1 test in 3.616s 2022-03-17T21:22:06.4111038Z 2022-03-17T21:22:06.4111269Z OK (skipped=1) 2022-03-17T21:22:06.4111613Z 2022-03-17T21:22:06.4111881Z Generating XML reports... 2022-03-17T21:22:06.4185147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212202.xml 2022-03-17T21:22:08.9990742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:09.0003703Z 2022-03-17T21:22:09.0003948Z Running tests... 2022-03-17T21:22:09.0004378Z ---------------------------------------------------------------------- 2022-03-17T21:22:09.6034074Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48771 2022-03-17T21:22:09.6065553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48772 2022-03-17T21:22:09.6066921Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:10.7628141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:10.7630098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:10.7631433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:10.7633098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:10.7736655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:10.8647712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:12.5155037Z skip (3.515s) 2022-03-17T21:22:12.5155437Z 2022-03-17T21:22:12.5156283Z ---------------------------------------------------------------------- 2022-03-17T21:22:12.5157071Z Ran 1 test in 3.515s 2022-03-17T21:22:12.5157432Z 2022-03-17T21:22:12.5159379Z OK (skipped=1) 2022-03-17T21:22:12.5159722Z 2022-03-17T21:22:12.5160011Z Generating XML reports... 2022-03-17T21:22:12.5238967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212208.xml 2022-03-17T21:22:15.3353617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:15.3368177Z 2022-03-17T21:22:15.3368464Z Running tests... 2022-03-17T21:22:15.3369070Z ---------------------------------------------------------------------- 2022-03-17T21:22:15.9404946Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48981 2022-03-17T21:22:15.9445927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48982 2022-03-17T21:22:15.9447318Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:17.0968615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:17.0970521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:17.0972507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:17.1073487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:17.1079420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:17.1994287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:18.8551649Z skip (3.518s) 2022-03-17T21:22:18.8552050Z 2022-03-17T21:22:18.8553013Z ---------------------------------------------------------------------- 2022-03-17T21:22:18.8553768Z Ran 1 test in 3.519s 2022-03-17T21:22:18.8554141Z 2022-03-17T21:22:18.8554370Z OK (skipped=1) 2022-03-17T21:22:18.8554734Z 2022-03-17T21:22:18.8554998Z Generating XML reports... 2022-03-17T21:22:18.8636102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212215.xml 2022-03-17T21:22:21.5121569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:21.5135698Z 2022-03-17T21:22:21.5136461Z Running tests... 2022-03-17T21:22:21.5137643Z ---------------------------------------------------------------------- 2022-03-17T21:22:22.1202116Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49191 2022-03-17T21:22:22.1237981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49192 2022-03-17T21:22:22.1239304Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:23.2765213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:23.2769263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:23.2770113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:23.2867171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:23.2869457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:23.3787705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:25.0321191Z skip (3.518s) 2022-03-17T21:22:25.0321585Z 2022-03-17T21:22:25.0323270Z ---------------------------------------------------------------------- 2022-03-17T21:22:25.0324750Z Ran 1 test in 3.519s 2022-03-17T21:22:25.0325122Z 2022-03-17T21:22:25.0325359Z OK (skipped=1) 2022-03-17T21:22:25.0325738Z 2022-03-17T21:22:25.0326014Z Generating XML reports... 2022-03-17T21:22:25.0407568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212221.xml 2022-03-17T21:22:27.6777431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:27.6790431Z 2022-03-17T21:22:27.6791023Z Running tests... 2022-03-17T21:22:27.6792508Z ---------------------------------------------------------------------- 2022-03-17T21:22:28.2779190Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49401 2022-03-17T21:22:28.2825419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49402 2022-03-17T21:22:28.2827066Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:29.4197723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:29.4200722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:29.4203428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:29.4206072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:29.4304357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:29.5234428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:31.1903182Z skip (3.511s) 2022-03-17T21:22:31.1903575Z 2022-03-17T21:22:31.1904565Z ---------------------------------------------------------------------- 2022-03-17T21:22:31.1905358Z Ran 1 test in 3.511s 2022-03-17T21:22:31.1905749Z 2022-03-17T21:22:31.1905961Z OK (skipped=1) 2022-03-17T21:22:31.1906319Z 2022-03-17T21:22:31.1906581Z Generating XML reports... 2022-03-17T21:22:31.1984407Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212227.xml 2022-03-17T21:22:33.7671826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:33.7685642Z 2022-03-17T21:22:33.7685862Z Running tests... 2022-03-17T21:22:33.7686539Z ---------------------------------------------------------------------- 2022-03-17T21:22:34.3731937Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49611 2022-03-17T21:22:34.3780022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49612 2022-03-17T21:22:34.3781346Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:35.5232438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:35.5234601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:35.5235908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:35.5237569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:35.5252880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:35.6246865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:37.1867017Z skip (3.418s) 2022-03-17T21:22:37.1867543Z 2022-03-17T21:22:37.1868600Z ---------------------------------------------------------------------- 2022-03-17T21:22:37.1869468Z Ran 1 test in 3.418s 2022-03-17T21:22:37.1869857Z 2022-03-17T21:22:37.1870107Z OK (skipped=1) 2022-03-17T21:22:37.1870481Z 2022-03-17T21:22:37.1870773Z Generating XML reports... 2022-03-17T21:22:37.1945863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212233.xml 2022-03-17T21:22:39.7758127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:39.7771407Z 2022-03-17T21:22:39.7771793Z Running tests... 2022-03-17T21:22:39.7773285Z ---------------------------------------------------------------------- 2022-03-17T21:22:40.4134403Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49821 2022-03-17T21:22:40.4188339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49822 2022-03-17T21:22:40.4189962Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:41.5640958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:41.5642134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:41.5642748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:41.5643514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:41.5752396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:41.6662517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:43.7276346Z skip (3.950s) 2022-03-17T21:22:43.7276827Z 2022-03-17T21:22:43.7277927Z ---------------------------------------------------------------------- 2022-03-17T21:22:43.7278861Z Ran 1 test in 3.950s 2022-03-17T21:22:43.7279213Z 2022-03-17T21:22:43.7279450Z OK (skipped=1) 2022-03-17T21:22:43.7279800Z 2022-03-17T21:22:43.7280493Z Generating XML reports... 2022-03-17T21:22:43.7342586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212239.xml 2022-03-17T21:22:46.3213060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:46.3226792Z 2022-03-17T21:22:46.3227225Z Running tests... 2022-03-17T21:22:46.9301884Z ---------------------------------------------------------------------- 2022-03-17T21:22:46.9302635Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50031 2022-03-17T21:22:46.9337775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50032 2022-03-17T21:22:46.9339109Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:48.0714178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:48.0716341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:48.0717669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:48.0719356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:48.0724362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:48.1727811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:49.8419961Z skip (3.519s) 2022-03-17T21:22:49.8420521Z 2022-03-17T21:22:49.8421515Z ---------------------------------------------------------------------- 2022-03-17T21:22:49.8422304Z Ran 1 test in 3.519s 2022-03-17T21:22:49.8422677Z 2022-03-17T21:22:49.8422911Z OK (skipped=1) 2022-03-17T21:22:49.8423263Z 2022-03-17T21:22:49.8423533Z Generating XML reports... 2022-03-17T21:22:49.8504844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212246.xml 2022-03-17T21:22:52.4709182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:52.4722656Z 2022-03-17T21:22:52.4722943Z Running tests... 2022-03-17T21:22:52.4724130Z ---------------------------------------------------------------------- 2022-03-17T21:22:53.0706259Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50241 2022-03-17T21:22:53.0743766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50242 2022-03-17T21:22:53.0745115Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:22:54.2177111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:54.2179198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:54.2180491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:54.2182180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:54.2285712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:54.3201338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:55.9845363Z skip (3.512s) 2022-03-17T21:22:55.9845768Z 2022-03-17T21:22:55.9847475Z ---------------------------------------------------------------------- 2022-03-17T21:22:55.9848277Z Ran 1 test in 3.512s 2022-03-17T21:22:55.9848645Z 2022-03-17T21:22:55.9848853Z OK (skipped=1) 2022-03-17T21:22:55.9849203Z 2022-03-17T21:22:55.9849468Z Generating XML reports... 2022-03-17T21:22:55.9927531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212252.xml 2022-03-17T21:22:58.6423963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:58.6436857Z 2022-03-17T21:22:58.6437137Z Running tests... 2022-03-17T21:22:58.6437671Z ---------------------------------------------------------------------- 2022-03-17T21:22:59.2503463Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50451 2022-03-17T21:22:59.2542633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50452 2022-03-17T21:22:59.2543993Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:23:00.4067485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:00.4068553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:00.4070306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:00.4169603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:00.4179167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:00.5182220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:02.1635420Z skip (3.519s) 2022-03-17T21:23:02.1635798Z 2022-03-17T21:23:02.1637143Z ---------------------------------------------------------------------- 2022-03-17T21:23:02.1638098Z Ran 1 test in 3.520s 2022-03-17T21:23:02.1638517Z 2022-03-17T21:23:02.1638785Z OK (skipped=1) 2022-03-17T21:23:02.1639176Z 2022-03-17T21:23:02.1639487Z Generating XML reports... 2022-03-17T21:23:02.1706922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212258.xml 2022-03-17T21:23:04.9378824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:04.9389249Z 2022-03-17T21:23:04.9390066Z Running tests... 2022-03-17T21:23:04.9391234Z ---------------------------------------------------------------------- 2022-03-17T21:23:05.5680481Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50661 2022-03-17T21:23:05.5739282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50662 2022-03-17T21:23:05.5740608Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:23:06.7322426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:06.7324535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:06.7325899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:06.7327601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:06.7438495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:06.8342357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:08.4823158Z skip (3.543s) 2022-03-17T21:23:08.4823670Z 2022-03-17T21:23:08.4825026Z ---------------------------------------------------------------------- 2022-03-17T21:23:08.4825982Z Ran 1 test in 3.543s 2022-03-17T21:23:08.4826511Z 2022-03-17T21:23:08.4826776Z OK (skipped=1) 2022-03-17T21:23:08.4828129Z 2022-03-17T21:23:08.4828500Z Generating XML reports... 2022-03-17T21:23:08.4907041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212304.xml 2022-03-17T21:23:11.1284605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:11.1303892Z 2022-03-17T21:23:11.1304249Z Running tests... 2022-03-17T21:23:11.1305445Z ---------------------------------------------------------------------- 2022-03-17T21:23:11.7379721Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50871 2022-03-17T21:23:11.7486498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50872 2022-03-17T21:23:11.7487668Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:23:12.8765485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:12.8766744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:12.8769368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:12.8869901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:12.8871220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:12.9788818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:22.5748121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.5749404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.5999718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.6001002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.6201092Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:23:22.6205560Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:23:22.6761597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.6763156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.7001406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.7003703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.7161572Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:23:22.7164060Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:23:22.7167363Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:23:22.7169553Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:23:22.7250837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.7252086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.7485997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:22.7487254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:25.4826680Z ok (14.352s) 2022-03-17T21:23:25.4827371Z 2022-03-17T21:23:25.4828912Z ---------------------------------------------------------------------- 2022-03-17T21:23:25.4829914Z Ran 1 test in 14.352s 2022-03-17T21:23:25.4830312Z 2022-03-17T21:23:25.4830501Z OK 2022-03-17T21:23:25.4830797Z 2022-03-17T21:23:25.4831055Z Generating XML reports... 2022-03-17T21:23:25.4907536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212311.xml 2022-03-17T21:23:28.5458366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:28.5469189Z 2022-03-17T21:23:28.5470195Z Running tests... 2022-03-17T21:23:28.5471407Z ---------------------------------------------------------------------- 2022-03-17T21:23:29.1788865Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51085 2022-03-17T21:23:29.1873856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51086 2022-03-17T21:23:29.1875242Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:23:30.3471591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:30.3473073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:30.3475161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:30.3476941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:30.3581127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:30.4485042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:42.6213814Z ok (14.074s) 2022-03-17T21:23:42.6214456Z 2022-03-17T21:23:42.6215869Z ---------------------------------------------------------------------- 2022-03-17T21:23:42.6216699Z Ran 1 test in 14.074s 2022-03-17T21:23:42.6217117Z 2022-03-17T21:23:42.6217390Z OK 2022-03-17T21:23:42.6217787Z 2022-03-17T21:23:42.6218188Z Generating XML reports... 2022-03-17T21:23:42.6320611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212328.xml 2022-03-17T21:23:45.4125155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:45.4140048Z 2022-03-17T21:23:45.4140998Z Running tests... 2022-03-17T21:23:45.4141707Z ---------------------------------------------------------------------- 2022-03-17T21:23:46.0224737Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51295 2022-03-17T21:23:46.0262192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51296 2022-03-17T21:23:46.0263482Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:23:47.1710701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:47.1722760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:47.1723479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:47.1812539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:47.1815932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:47.2732517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:56.7050872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:56.7054810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:59.3596821Z ok (13.945s) 2022-03-17T21:23:59.3597214Z 2022-03-17T21:23:59.3598077Z ---------------------------------------------------------------------- 2022-03-17T21:23:59.3598843Z Ran 1 test in 13.946s 2022-03-17T21:23:59.3599222Z 2022-03-17T21:23:59.3599405Z OK 2022-03-17T21:23:59.3599699Z 2022-03-17T21:23:59.3599972Z Generating XML reports... 2022-03-17T21:23:59.3681550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212345.xml 2022-03-17T21:24:01.9654212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:01.9668641Z 2022-03-17T21:24:01.9668935Z Running tests... 2022-03-17T21:24:01.9669384Z ---------------------------------------------------------------------- 2022-03-17T21:24:02.5720788Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51509 2022-03-17T21:24:02.5767694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51510 2022-03-17T21:24:02.5769026Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:24:03.6946259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:03.6946829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:03.6947691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:03.6948497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:03.6949097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:03.6952010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:03.7125087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:03.7126568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:05.5858749Z ok (3.619s) 2022-03-17T21:24:05.5859164Z 2022-03-17T21:24:05.5860568Z ---------------------------------------------------------------------- 2022-03-17T21:24:05.5861345Z Ran 1 test in 3.619s 2022-03-17T21:24:05.5861708Z 2022-03-17T21:24:05.5861907Z OK 2022-03-17T21:24:05.5862195Z 2022-03-17T21:24:05.5862476Z Generating XML reports... 2022-03-17T21:24:05.5937497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212401.xml 2022-03-17T21:24:08.2898736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:08.2908417Z 2022-03-17T21:24:08.2909954Z Running tests... 2022-03-17T21:24:08.2911158Z ---------------------------------------------------------------------- 2022-03-17T21:24:08.8919089Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51785 2022-03-17T21:24:08.9011433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51786 2022-03-17T21:24:08.9016667Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:24:10.0357942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:10.0360252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:10.0362306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:10.0459680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:10.0477671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:10.1469176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:19.9576097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:19.9580177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:22.7341718Z ok (14.443s) 2022-03-17T21:24:22.7342098Z 2022-03-17T21:24:22.7343014Z ---------------------------------------------------------------------- 2022-03-17T21:24:22.7343844Z Ran 1 test in 14.443s 2022-03-17T21:24:22.7344218Z 2022-03-17T21:24:22.7344384Z OK 2022-03-17T21:24:22.7344685Z 2022-03-17T21:24:22.7344959Z Generating XML reports... 2022-03-17T21:24:22.7427395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212408.xml 2022-03-17T21:24:25.5384804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:25.5396699Z 2022-03-17T21:24:25.5396959Z Running tests... 2022-03-17T21:24:25.5398568Z ---------------------------------------------------------------------- 2022-03-17T21:24:26.1450233Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51999 2022-03-17T21:24:26.1512353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52000 2022-03-17T21:24:26.1513662Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:24:27.3057839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:27.3060225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:27.3061745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:27.3063950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:27.3073368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:27.4071690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:29.1619476Z skip (3.622s) 2022-03-17T21:24:29.1620072Z 2022-03-17T21:24:29.1621192Z ---------------------------------------------------------------------- 2022-03-17T21:24:29.1622059Z Ran 1 test in 3.622s 2022-03-17T21:24:29.1622498Z 2022-03-17T21:24:29.1622808Z OK (skipped=1) 2022-03-17T21:24:29.1630536Z 2022-03-17T21:24:29.1631752Z Generating XML reports... 2022-03-17T21:24:29.1711095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212425.xml 2022-03-17T21:24:31.8943429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:31.8953795Z 2022-03-17T21:24:31.8954348Z Running tests... 2022-03-17T21:24:31.8955541Z ---------------------------------------------------------------------- 2022-03-17T21:24:32.5014889Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52209 2022-03-17T21:24:32.5053457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52210 2022-03-17T21:24:32.5055085Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:24:33.6571691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:33.6573934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:33.6575278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:33.6576698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:33.6684045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:33.7588820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:35.4155159Z skip (3.520s) 2022-03-17T21:24:35.4155754Z 2022-03-17T21:24:35.4156875Z ---------------------------------------------------------------------- 2022-03-17T21:24:35.4157831Z Ran 1 test in 3.520s 2022-03-17T21:24:35.4158292Z 2022-03-17T21:24:35.4158601Z OK (skipped=1) 2022-03-17T21:24:35.4159044Z 2022-03-17T21:24:35.4159334Z Generating XML reports... 2022-03-17T21:24:35.4228347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212431.xml 2022-03-17T21:24:38.2672218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:38.2683040Z 2022-03-17T21:24:38.2683780Z Running tests... 2022-03-17T21:24:38.2685276Z ---------------------------------------------------------------------- 2022-03-17T21:24:38.8926771Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52419 2022-03-17T21:24:38.8967187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52420 2022-03-17T21:24:38.8968774Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:24:40.0370700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:40.0372080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:40.0374528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:40.0376526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:40.0378044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:40.0379394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:52.4296573Z ok (14.161s) 2022-03-17T21:24:52.4296978Z 2022-03-17T21:24:52.4298394Z ---------------------------------------------------------------------- 2022-03-17T21:24:52.4299224Z Ran 1 test in 14.161s 2022-03-17T21:24:52.4299616Z 2022-03-17T21:24:52.4299827Z OK 2022-03-17T21:24:52.4300125Z 2022-03-17T21:24:52.4300407Z Generating XML reports... 2022-03-17T21:24:52.4382714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212438.xml 2022-03-17T21:24:55.1113298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:55.1123762Z 2022-03-17T21:24:55.1124443Z Running tests... 2022-03-17T21:24:55.1125869Z ---------------------------------------------------------------------- 2022-03-17T21:24:55.7169081Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52641 2022-03-17T21:24:55.7207923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52642 2022-03-17T21:24:55.7209951Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:24:56.8802631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:56.8804160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:56.8806286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:56.8808102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:56.8910832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:56.9814875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:09.6484421Z ok (14.535s) 2022-03-17T21:25:09.6484832Z 2022-03-17T21:25:09.6485822Z ---------------------------------------------------------------------- 2022-03-17T21:25:09.6489002Z Ran 1 test in 14.536s 2022-03-17T21:25:09.6489480Z 2022-03-17T21:25:09.6489717Z OK 2022-03-17T21:25:09.6490075Z 2022-03-17T21:25:09.6490413Z Generating XML reports... 2022-03-17T21:25:09.6565747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212455.xml 2022-03-17T21:25:12.2319634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:12.2332586Z 2022-03-17T21:25:12.2333160Z Running tests... 2022-03-17T21:25:12.2334588Z ---------------------------------------------------------------------- 2022-03-17T21:25:12.8407820Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52863 2022-03-17T21:25:12.8446027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52864 2022-03-17T21:25:12.8447644Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:25:13.9754150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:13.9755651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:13.9758275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:13.9760180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:13.9862949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:14.0765930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:26.3780757Z ok (14.144s) 2022-03-17T21:25:26.3781308Z 2022-03-17T21:25:26.3782500Z ---------------------------------------------------------------------- 2022-03-17T21:25:26.3783298Z Ran 1 test in 14.145s 2022-03-17T21:25:26.3783771Z 2022-03-17T21:25:26.3784039Z OK 2022-03-17T21:25:26.3788137Z 2022-03-17T21:25:26.3788829Z Generating XML reports... 2022-03-17T21:25:26.3861466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212512.xml 2022-03-17T21:25:29.2747354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:29.2759563Z 2022-03-17T21:25:29.2760788Z Running tests... 2022-03-17T21:25:29.2762684Z ---------------------------------------------------------------------- 2022-03-17T21:25:29.8944172Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53073 2022-03-17T21:25:29.8991467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53074 2022-03-17T21:25:29.8993599Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:25:31.0590841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:31.0591566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:31.0592066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:31.0592958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:31.0608760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:31.1602697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:40.4799943Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:40.5066395Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:40.6711454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:40.6712709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:43.3275806Z ok (14.051s) 2022-03-17T21:25:43.3282661Z 2022-03-17T21:25:43.3283829Z ---------------------------------------------------------------------- 2022-03-17T21:25:43.3285296Z Ran 1 test in 14.052s 2022-03-17T21:25:43.3285803Z 2022-03-17T21:25:43.3286042Z OK 2022-03-17T21:25:43.3286405Z 2022-03-17T21:25:43.3286737Z Generating XML reports... 2022-03-17T21:25:43.3363386Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212529.xml 2022-03-17T21:25:46.1584142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:46.1596400Z 2022-03-17T21:25:46.1597067Z Running tests... 2022-03-17T21:25:46.1598547Z ---------------------------------------------------------------------- 2022-03-17T21:25:46.7585758Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53287 2022-03-17T21:25:46.7624276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53288 2022-03-17T21:25:46.7625862Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:25:47.9147870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:47.9149376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:47.9151526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:47.9249665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:47.9255410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:47.9256848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:48.3379682Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:48.3382032Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:48.3385315Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:48.3387535Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:26:00.2893192Z ok (14.129s) 2022-03-17T21:26:00.2893784Z 2022-03-17T21:26:00.2895131Z ---------------------------------------------------------------------- 2022-03-17T21:26:00.2895912Z Ran 1 test in 14.130s 2022-03-17T21:26:00.2896300Z 2022-03-17T21:26:00.2896562Z OK 2022-03-17T21:26:00.2896967Z 2022-03-17T21:26:00.2897287Z Generating XML reports... 2022-03-17T21:26:00.2974263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212546.xml 2022-03-17T21:26:02.9197436Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:02.9211008Z 2022-03-17T21:26:02.9211544Z Running tests... 2022-03-17T21:26:02.9212740Z ---------------------------------------------------------------------- 2022-03-17T21:26:03.5428394Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53501 2022-03-17T21:26:03.5468298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53502 2022-03-17T21:26:03.5470070Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:26:04.6368470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:04.6369044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:04.6370132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:04.6370953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:04.6475602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:04.7383000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:14.3239500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:14.3240794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:14.3885052Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:26:14.3890807Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:26:17.1753468Z ok (14.254s) 2022-03-17T21:26:17.1753857Z 2022-03-17T21:26:17.1754900Z ---------------------------------------------------------------------- 2022-03-17T21:26:17.1755886Z Ran 1 test in 14.254s 2022-03-17T21:26:17.1756340Z 2022-03-17T21:26:17.1756584Z OK 2022-03-17T21:26:17.1756961Z 2022-03-17T21:26:17.1757339Z Generating XML reports... 2022-03-17T21:26:17.1828229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212602.xml 2022-03-17T21:26:19.8165090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:19.8175162Z 2022-03-17T21:26:19.8175776Z Running tests... 2022-03-17T21:26:19.8177223Z ---------------------------------------------------------------------- 2022-03-17T21:26:20.4117251Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.594s) 2022-03-17T21:26:20.4118154Z 2022-03-17T21:26:20.4119354Z ---------------------------------------------------------------------- 2022-03-17T21:26:20.4120546Z Ran 1 test in 0.594s 2022-03-17T21:26:20.4120932Z 2022-03-17T21:26:20.4121213Z OK (skipped=1) 2022-03-17T21:26:20.4121681Z 2022-03-17T21:26:20.4122261Z Generating XML reports... 2022-03-17T21:26:20.4148262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212619.xml 2022-03-17T21:26:22.9861480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:22.9877096Z 2022-03-17T21:26:22.9877431Z Running tests... 2022-03-17T21:26:22.9877881Z ---------------------------------------------------------------------- 2022-03-17T21:26:23.5932064Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53788 2022-03-17T21:26:23.5980361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53789 2022-03-17T21:26:23.5981725Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:26:24.7291235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:24.7292293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:24.7293863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:24.7394271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:24.7401509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:24.8305714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:27.2089233Z ok (4.221s) 2022-03-17T21:26:27.2089562Z 2022-03-17T21:26:27.2090320Z ---------------------------------------------------------------------- 2022-03-17T21:26:27.2091310Z Ran 1 test in 4.221s 2022-03-17T21:26:27.2091635Z 2022-03-17T21:26:27.2091791Z OK 2022-03-17T21:26:27.2092083Z 2022-03-17T21:26:27.2092358Z Generating XML reports... 2022-03-17T21:26:27.2172083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212622.xml 2022-03-17T21:26:29.8699450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:29.8710335Z 2022-03-17T21:26:29.8710624Z Running tests... 2022-03-17T21:26:29.8711063Z ---------------------------------------------------------------------- 2022-03-17T21:26:30.4775487Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53998 2022-03-17T21:26:30.4818181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53999 2022-03-17T21:26:30.4819375Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:26:31.6472982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:31.6474927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:31.6476225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:31.6477891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:31.6492103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:31.7491163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:32.1540051Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:26:32.1542349Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:26:32.1546075Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:26:32.1548321Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:26:41.3925964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:41.3927448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:41.4065753Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:26:41.4070422Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:26:41.4287512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:41.4289011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:41.4414546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:41.4416005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:44.1084052Z ok (14.237s) 2022-03-17T21:26:44.1084698Z 2022-03-17T21:26:44.1086073Z ---------------------------------------------------------------------- 2022-03-17T21:26:44.1087001Z Ran 1 test in 14.237s 2022-03-17T21:26:44.1087372Z 2022-03-17T21:26:44.1087541Z OK 2022-03-17T21:26:44.1087861Z 2022-03-17T21:26:44.1096291Z Generating XML reports... 2022-03-17T21:26:44.1166172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212629.xml 2022-03-17T21:26:46.8848167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:46.8858167Z 2022-03-17T21:26:46.8858562Z Running tests... 2022-03-17T21:26:46.8859004Z ---------------------------------------------------------------------- 2022-03-17T21:26:47.4949361Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54212 2022-03-17T21:26:47.4990289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54213 2022-03-17T21:26:47.4991432Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:26:48.6049414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:48.6049981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:48.6050867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:48.6051546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:48.6053451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:48.6054016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:49.0207196Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:26:49.0209584Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:26:49.0212886Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:26:49.0215066Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:26:58.2167106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:58.2168358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:00.8273518Z ok (13.941s) 2022-03-17T21:27:00.8274076Z 2022-03-17T21:27:00.8275467Z ---------------------------------------------------------------------- 2022-03-17T21:27:00.8276623Z Ran 1 test in 13.941s 2022-03-17T21:27:00.8277087Z 2022-03-17T21:27:00.8277854Z OK 2022-03-17T21:27:00.8278230Z 2022-03-17T21:27:00.8278577Z Generating XML reports... 2022-03-17T21:27:00.8345468Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212646.xml 2022-03-17T21:27:03.4131056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:27:03.4144645Z 2022-03-17T21:27:03.4144967Z Running tests... 2022-03-17T21:27:03.4145644Z ---------------------------------------------------------------------- 2022-03-17T21:27:04.0225327Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54426 2022-03-17T21:27:04.0259678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54427 2022-03-17T21:27:04.0261024Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:27:05.1740709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:27:05.1741814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:05.1742322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:27:05.1742965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:05.1851502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:27:05.2766982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:27:14.8407055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:14.8408543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:14.9251469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:14.9252523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:17.6600035Z ok (14.245s) 2022-03-17T21:27:17.6600444Z 2022-03-17T21:27:17.6601676Z ---------------------------------------------------------------------- 2022-03-17T21:27:17.6602888Z Ran 1 test in 14.246s 2022-03-17T21:27:17.6603332Z 2022-03-17T21:27:17.6603539Z OK 2022-03-17T21:27:17.6604480Z 2022-03-17T21:27:17.6604789Z Generating XML reports... 2022-03-17T21:27:17.6684218Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212703.xml 2022-03-17T21:27:20.2468150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:27:20.2479668Z 2022-03-17T21:27:20.2479921Z Running tests... 2022-03-17T21:27:20.2480472Z ---------------------------------------------------------------------- 2022-03-17T21:27:20.8430711Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... skip (0.595s) 2022-03-17T21:27:20.8431555Z 2022-03-17T21:27:20.8432353Z ---------------------------------------------------------------------- 2022-03-17T21:27:20.8433166Z Ran 1 test in 0.595s 2022-03-17T21:27:20.8433589Z 2022-03-17T21:27:20.8433905Z OK (skipped=1) 2022-03-17T21:27:20.8434348Z 2022-03-17T21:27:20.8434691Z Generating XML reports... 2022-03-17T21:27:20.8462710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212720.xml 2022-03-17T21:27:23.4411196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:27:23.4421395Z 2022-03-17T21:27:23.4422116Z Running tests... 2022-03-17T21:27:23.4423561Z ---------------------------------------------------------------------- 2022-03-17T21:27:24.0586084Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54707 2022-03-17T21:27:24.0634497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54708 2022-03-17T21:27:24.0636057Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:27:25.2181389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:27:25.2183761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:25.2185132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:27:25.2186841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:25.2291332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:27:25.3204021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:27:27.6732845Z ok (4.230s) 2022-03-17T21:27:27.6734429Z 2022-03-17T21:27:27.6735680Z ---------------------------------------------------------------------- 2022-03-17T21:27:27.6736656Z Ran 1 test in 4.231s 2022-03-17T21:27:27.6737127Z 2022-03-17T21:27:27.6737343Z OK 2022-03-17T21:27:27.6737747Z 2022-03-17T21:27:27.6738037Z Generating XML reports... 2022-03-17T21:27:27.6795459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212723.xml 2022-03-17T21:27:30.3342471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:27:30.3356839Z 2022-03-17T21:27:30.3356963Z Running tests... 2022-03-17T21:27:30.3357529Z ---------------------------------------------------------------------- 2022-03-17T21:27:30.9406662Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54917 2022-03-17T21:27:30.9448207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54918 2022-03-17T21:27:30.9449778Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:27:32.0927690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:27:32.0928526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:27:32.0929385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:32.0930194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:32.0944065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:27:32.0945162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:27:34.5554188Z ok (4.219s) 2022-03-17T21:27:34.5554568Z 2022-03-17T21:27:34.5555520Z ---------------------------------------------------------------------- 2022-03-17T21:27:34.5556395Z Ran 1 test in 4.220s 2022-03-17T21:27:34.5556829Z 2022-03-17T21:27:34.5557014Z OK 2022-03-17T21:27:34.5557315Z 2022-03-17T21:27:34.5557609Z Generating XML reports... 2022-03-17T21:27:34.5626129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212730.xml 2022-03-17T21:27:37.4886813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:27:37.4896722Z 2022-03-17T21:27:37.4897185Z Running tests... 2022-03-17T21:27:37.4898179Z ---------------------------------------------------------------------- 2022-03-17T21:27:38.0987191Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55127 2022-03-17T21:27:38.1022840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55128 2022-03-17T21:27:38.1024197Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:27:39.2372978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:27:39.2373718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:39.2374217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:27:39.2374858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:39.2390072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:27:39.3386147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:27:48.8600076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:48.8601350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:51.5264633Z ok (14.036s) 2022-03-17T21:27:51.5265021Z 2022-03-17T21:27:51.5266034Z ---------------------------------------------------------------------- 2022-03-17T21:27:51.5266837Z Ran 1 test in 14.037s 2022-03-17T21:27:51.5267198Z 2022-03-17T21:27:51.5267386Z OK 2022-03-17T21:27:51.5267685Z 2022-03-17T21:27:51.5267975Z Generating XML reports... 2022-03-17T21:27:51.5347838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212737.xml 2022-03-17T21:27:54.2966087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:27:54.2976271Z 2022-03-17T21:27:54.2976377Z Running tests... 2022-03-17T21:27:54.2976965Z ---------------------------------------------------------------------- 2022-03-17T21:27:54.9086820Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55341 2022-03-17T21:27:54.9127100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55342 2022-03-17T21:27:54.9128901Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:27:56.0216132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:27:56.0216725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:27:56.0217687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:56.0218554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:27:56.0222694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:27:56.0223220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:27:56.3678638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:27:56.3771465Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:27:56.3772708Z Running test: batch_norm_net sync interval 2022-03-17T21:27:56.3773063Z 1 with iteration mapping 2022-03-17T21:27:56.3773343Z {0: 0, 1: 2} 2022-03-17T21:28:05.7597397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:05.7687166Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:05.7689225Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:05.7692008Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:05.7693943Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:05.9416653Z Running test: batch_norm_net sync interval 2022-03-17T21:28:05.9417541Z 1 with iteration mapping 2022-03-17T21:28:05.9418266Z {0: 0, 1: 3} 2022-03-17T21:28:05.9525024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:05.9640713Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:05.9642566Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:05.9645607Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:05.9648258Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:05.9766033Z Running test: batch_norm_net sync interval 2022-03-17T21:28:05.9766857Z 1 with iteration mapping 2022-03-17T21:28:05.9767570Z {0: 0, 1: 10} 2022-03-17T21:28:05.9865897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:06.0293112Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:06.0294636Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:06.0297319Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:06.0299404Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:06.0422626Z Running test: batch_norm_net sync interval 2022-03-17T21:28:06.0423506Z 1 with iteration mapping 2022-03-17T21:28:06.0424713Z {0: 5, 1: 7} 2022-03-17T21:28:15.1280288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:15.1286594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:15.1593454Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:15.1595244Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:15.1598219Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:15.1600354Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:15.1736400Z Running test: batch_norm_net sync interval 2022-03-17T21:28:15.1736970Z 1 with iteration mapping 2022-03-17T21:28:15.1737378Z {0: 5, 1: 8} 2022-03-17T21:28:15.1827579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:15.1828938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:15.2157148Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:15.2159414Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:15.2162508Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:15.2165280Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:15.2286810Z Running test: batch_norm_net sync interval 2022-03-17T21:28:15.2287351Z 1 with iteration mapping 2022-03-17T21:28:15.2287800Z {0: 5, 1: 15} 2022-03-17T21:28:15.2364410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:15.2365763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:15.3002762Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:15.3004294Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:15.3007108Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:15.3009196Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:15.3128316Z Running test: large_conv_model sync interval 2022-03-17T21:28:15.3129148Z 1 with iteration mapping 2022-03-17T21:28:15.3129875Z {0: 0, 1: 2} 2022-03-17T21:28:15.8112629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.0011384Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.0014160Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.0017701Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.0019778Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.0312522Z Running test: large_conv_model sync interval 2022-03-17T21:28:16.0312908Z 1 with iteration mapping 2022-03-17T21:28:16.0313180Z {0: 0, 1: 3} 2022-03-17T21:28:16.2236091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.5966222Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.5968357Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.5971288Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.5973758Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.6205102Z Running test: large_conv_model sync interval 2022-03-17T21:28:16.6205523Z 1 with iteration mapping 2022-03-17T21:28:16.6205801Z {0: 0, 1: 10} 2022-03-17T21:28:16.8141900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:18.4971482Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:18.4974552Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:18.4977534Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:18.4979521Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:18.5186709Z Running test: large_conv_model sync interval 2022-03-17T21:28:18.5187359Z 1 with iteration mapping 2022-03-17T21:28:18.5187883Z {0: 5, 1: 7} 2022-03-17T21:28:19.0087878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:19.0088446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:20.1248865Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:20.1251041Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:20.1254007Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:20.1256069Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:20.1479322Z Running test: large_conv_model sync interval 2022-03-17T21:28:20.1479690Z 1 with iteration mapping 2022-03-17T21:28:20.1480189Z {0: 5, 1: 8} 2022-03-17T21:28:20.3397215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:20.3398217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:21.6501607Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:21.6503121Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:21.6505246Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:21.6507179Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:21.6749241Z Running test: large_conv_model sync interval 2022-03-17T21:28:21.6749845Z 1 with iteration mapping 2022-03-17T21:28:21.6750271Z {0: 5, 1: 15} 2022-03-17T21:28:21.8646849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:21.8647702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.4570117Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.4572458Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.4575480Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.4577569Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.4785341Z Running test: small_model sync interval 2022-03-17T21:28:24.4786507Z 1 with iteration mapping 2022-03-17T21:28:24.4787005Z {0: 0, 1: 2} 2022-03-17T21:28:24.4962788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.5253553Z Running test: small_model sync interval 2022-03-17T21:28:24.5254179Z 1 with iteration mapping 2022-03-17T21:28:24.5254653Z {0: 0, 1: 3} 2022-03-17T21:28:24.5393915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.5681338Z Running test: small_model sync interval 2022-03-17T21:28:24.5682431Z 1 with iteration mapping 2022-03-17T21:28:24.5683117Z {0: 0, 1: 10} 2022-03-17T21:28:24.5805831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.6539528Z Running test: small_model sync interval 2022-03-17T21:28:24.6539917Z 1 with iteration mapping 2022-03-17T21:28:24.6540218Z {0: 5, 1: 7} 2022-03-17T21:28:24.6647963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.6649323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.7127349Z Running test: small_model sync interval 2022-03-17T21:28:24.7128204Z 1 with iteration mapping 2022-03-17T21:28:24.7128895Z {0: 5, 1: 8} 2022-03-17T21:28:24.7243225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.7244536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.7798562Z Running test: small_model sync interval 2022-03-17T21:28:24.7799394Z 1 with iteration mapping 2022-03-17T21:28:24.7800046Z {0: 5, 1: 15} 2022-03-17T21:28:24.8046855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.8048149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.9081681Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:24.9082829Z 1 with iteration mapping 2022-03-17T21:28:24.9083466Z {0: 0, 1: 2} 2022-03-17T21:28:24.9186954Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9189147Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9192049Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9193980Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9228401Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:24.9229314Z 1 with iteration mapping 2022-03-17T21:28:24.9229952Z {0: 0, 1: 3} 2022-03-17T21:28:24.9354342Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9356468Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9359908Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9362192Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9394047Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:24.9394943Z 1 with iteration mapping 2022-03-17T21:28:24.9395602Z {0: 0, 1: 10} 2022-03-17T21:28:24.9684693Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9686826Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9689716Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9691773Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9722934Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:24.9723870Z 1 with iteration mapping 2022-03-17T21:28:24.9724528Z {0: 5, 1: 7} 2022-03-17T21:28:24.9765714Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:28:24.9950931Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9953088Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9956076Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:24.9958123Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:24.9992322Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:24.9993238Z 1 with iteration mapping 2022-03-17T21:28:24.9993872Z {0: 5, 1: 8} 2022-03-17T21:28:25.0229294Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0231735Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0234623Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0236706Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0268274Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.0269181Z 1 with iteration mapping 2022-03-17T21:28:25.0269850Z {0: 5, 1: 15} 2022-03-17T21:28:25.0652163Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0654299Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0657201Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0659241Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0693067Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.0694013Z 1 with iteration mapping 2022-03-17T21:28:25.0694672Z {0: 0, 1: 2} 2022-03-17T21:28:25.0734691Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:28:25.0804989Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0807139Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0810051Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0812080Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0843711Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.0844621Z 1 with iteration mapping 2022-03-17T21:28:25.0845260Z {0: 0, 1: 3} 2022-03-17T21:28:25.0979278Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0981737Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.0984701Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.0986744Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1018704Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.1019597Z 1 with iteration mapping 2022-03-17T21:28:25.1020253Z {0: 0, 1: 10} 2022-03-17T21:28:25.1358060Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.1360202Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1363269Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.1365318Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1405956Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.1406909Z 1 with iteration mapping 2022-03-17T21:28:25.1407588Z {0: 5, 1: 7} 2022-03-17T21:28:25.1629661Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.1632113Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1635063Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.1637106Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1674857Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.1675772Z 1 with iteration mapping 2022-03-17T21:28:25.1676417Z {0: 5, 1: 8} 2022-03-17T21:28:25.1903757Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.1905904Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1908829Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.1911199Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.1945456Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:25.1946340Z 1 with iteration mapping 2022-03-17T21:28:25.1947006Z {0: 5, 1: 15} 2022-03-17T21:28:25.2343256Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.2345408Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.2348293Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:25.2350341Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:25.2386244Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:25.2387127Z 1 with iteration mapping 2022-03-17T21:28:25.2387788Z {0: 0, 1: 2} 2022-03-17T21:28:25.2443192Z registered hook 2022-03-17T21:28:25.2444634Z registered hook 2022-03-17T21:28:25.2522971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.2727923Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:25.2728769Z 1 with iteration mapping 2022-03-17T21:28:25.2753872Z {0: 0, 1: 3} 2022-03-17T21:28:25.2777617Z registered hook 2022-03-17T21:28:25.2778558Z registered hook 2022-03-17T21:28:25.2849108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.3137339Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:25.3138606Z 1 with iteration mapping 2022-03-17T21:28:25.3139233Z {0: 0, 1: 10} 2022-03-17T21:28:25.3177863Z registered hook 2022-03-17T21:28:25.3178848Z registered hook 2022-03-17T21:28:25.3254628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.4040088Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:25.4041011Z 1 with iteration mapping 2022-03-17T21:28:25.4041727Z {0: 5, 1: 7} 2022-03-17T21:28:25.4083046Z registered hook 2022-03-17T21:28:25.4084041Z registered hook 2022-03-17T21:28:25.4165241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.4166573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.4732588Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:25.4733452Z 1 with iteration mapping 2022-03-17T21:28:25.4734115Z {0: 5, 1: 8} 2022-03-17T21:28:25.4773719Z registered hook 2022-03-17T21:28:25.4774737Z registered hook 2022-03-17T21:28:25.4851487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.4853040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.5514773Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:25.5515380Z 1 with iteration mapping 2022-03-17T21:28:25.5515823Z {0: 5, 1: 15} 2022-03-17T21:28:25.5563315Z registered hook 2022-03-17T21:28:25.5564033Z registered hook 2022-03-17T21:28:25.5637923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.5638789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.6661133Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:25.6662054Z 1 with iteration mapping 2022-03-17T21:28:25.6662701Z {0: 0, 1: 2} 2022-03-17T21:28:25.6707790Z registered hook 2022-03-17T21:28:25.6708778Z registered hook 2022-03-17T21:28:25.6728181Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T21:28:25.6733775Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T21:28:25.6800825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.6808796Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T21:28:25.6812890Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T21:28:25.7424311Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:25.7425100Z 1 with iteration mapping 2022-03-17T21:28:25.7426046Z {0: 0, 1: 3} 2022-03-17T21:28:25.7464631Z registered hook 2022-03-17T21:28:25.7465372Z registered hook 2022-03-17T21:28:25.7522865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.7756335Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:25.7757815Z 1 with iteration mapping 2022-03-17T21:28:25.7758457Z {0: 0, 1: 10} 2022-03-17T21:28:25.7796081Z registered hook 2022-03-17T21:28:25.7797023Z registered hook 2022-03-17T21:28:25.7857346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.8373508Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:25.8374457Z 1 with iteration mapping 2022-03-17T21:28:25.8375109Z {0: 5, 1: 7} 2022-03-17T21:28:25.8423018Z registered hook 2022-03-17T21:28:25.8424014Z registered hook 2022-03-17T21:28:25.8483367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.8484295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.8887490Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:25.8888078Z 1 with iteration mapping 2022-03-17T21:28:25.8888544Z {0: 5, 1: 8} 2022-03-17T21:28:25.8937498Z registered hook 2022-03-17T21:28:25.8938200Z registered hook 2022-03-17T21:28:25.8998479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.8999843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:25.9428222Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:25.9428819Z 1 with iteration mapping 2022-03-17T21:28:25.9429266Z {0: 5, 1: 15} 2022-03-17T21:28:25.9478835Z registered hook 2022-03-17T21:28:25.9482452Z registered hook 2022-03-17T21:28:25.9547472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.0248148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.0248979Z Running test: batch_norm_net sync interval 2022-03-17T21:28:26.0249539Z 2 with iteration mapping 2022-03-17T21:28:26.0250037Z {0: 0, 1: 2} 2022-03-17T21:28:26.0359475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.0426136Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.0427628Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.0429570Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.0430923Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.0557980Z Running test: batch_norm_net sync interval 2022-03-17T21:28:26.0558776Z 2 with iteration mapping 2022-03-17T21:28:26.0559384Z {0: 0, 1: 3} 2022-03-17T21:28:26.0665317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.0771610Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.0773679Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.0776471Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.0778445Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.0910006Z Running test: batch_norm_net sync interval 2022-03-17T21:28:26.0910794Z 2 with iteration mapping 2022-03-17T21:28:26.0911440Z {0: 0, 1: 10} 2022-03-17T21:28:26.0997769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.1316192Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.1318582Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.1321384Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.1323588Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.1447697Z Running test: batch_norm_net sync interval 2022-03-17T21:28:26.1448461Z 2 with iteration mapping 2022-03-17T21:28:26.1449094Z {0: 5, 1: 7} 2022-03-17T21:28:26.1551737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.1552983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.1768561Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.1770622Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.1773428Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.1775400Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.1892838Z Running test: batch_norm_net sync interval 2022-03-17T21:28:26.1893604Z 2 with iteration mapping 2022-03-17T21:28:26.1894223Z {0: 5, 1: 8} 2022-03-17T21:28:26.2005720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.2006949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.2250652Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.2252773Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.2255565Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.2257565Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.2380606Z Running test: batch_norm_net sync interval 2022-03-17T21:28:26.2381599Z 2 with iteration mapping 2022-03-17T21:28:26.2382420Z {0: 5, 1: 15} 2022-03-17T21:28:26.2468694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.2470165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.2928605Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.2931103Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.2933946Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.2935940Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.3057828Z Running test: large_conv_model sync interval 2022-03-17T21:28:26.3058837Z 3 with iteration mapping 2022-03-17T21:28:26.3059606Z {0: 0, 1: 2} 2022-03-17T21:28:26.4983384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:26.6844856Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.6847029Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.6849807Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:26.6851759Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:26.7060256Z Running test: large_conv_model sync interval 2022-03-17T21:28:26.7061301Z 3 with iteration mapping 2022-03-17T21:28:26.7062107Z {0: 0, 1: 3} 2022-03-17T21:28:26.8978676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:27.2657726Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:27.2659775Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:27.2662564Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:27.2664521Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:27.2880088Z Running test: large_conv_model sync interval 2022-03-17T21:28:27.2880606Z 3 with iteration mapping 2022-03-17T21:28:27.2880974Z {0: 0, 1: 10} 2022-03-17T21:28:27.4792284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:29.1416779Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:29.1418898Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:29.1422065Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:29.1424006Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:29.1609750Z Running test: large_conv_model sync interval 2022-03-17T21:28:29.1610625Z 3 with iteration mapping 2022-03-17T21:28:29.1611362Z {0: 5, 1: 7} 2022-03-17T21:28:29.3507759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:29.3508917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:30.4530284Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:30.4532348Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:30.4535119Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:30.4537058Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:30.4763992Z Running test: large_conv_model sync interval 2022-03-17T21:28:30.4764494Z 3 with iteration mapping 2022-03-17T21:28:30.4764863Z {0: 5, 1: 8} 2022-03-17T21:28:30.6689305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:30.6690006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:31.9680295Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:31.9682688Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:31.9685510Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:31.9687465Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:31.9885682Z Running test: large_conv_model sync interval 2022-03-17T21:28:31.9886496Z 3 with iteration mapping 2022-03-17T21:28:31.9887126Z {0: 5, 1: 15} 2022-03-17T21:28:32.1803389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:32.1804614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.7489562Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:34.7491930Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:34.7494681Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:34.7496612Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:34.7696664Z Running test: small_model sync interval 2022-03-17T21:28:34.7696984Z 4 with iteration mapping 2022-03-17T21:28:34.7697402Z {0: 0, 1: 2} 2022-03-17T21:28:34.7809769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.7997306Z Running test: small_model sync interval 2022-03-17T21:28:34.7997620Z 4 with iteration mapping 2022-03-17T21:28:34.7997851Z {0: 0, 1: 3} 2022-03-17T21:28:34.8113299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.8320133Z Running test: small_model sync interval 2022-03-17T21:28:34.8321088Z 4 with iteration mapping 2022-03-17T21:28:34.8322101Z {0: 0, 1: 10} 2022-03-17T21:28:34.8452628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.8890765Z Running test: small_model sync interval 2022-03-17T21:28:34.8891791Z 4 with iteration mapping 2022-03-17T21:28:34.8892607Z {0: 5, 1: 7} 2022-03-17T21:28:34.9015627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.9017207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.9315484Z Running test: small_model sync interval 2022-03-17T21:28:34.9316524Z 4 with iteration mapping 2022-03-17T21:28:34.9317317Z {0: 5, 1: 8} 2022-03-17T21:28:34.9433457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.9435021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.9754231Z Running test: small_model sync interval 2022-03-17T21:28:34.9755388Z 4 with iteration mapping 2022-03-17T21:28:34.9756045Z {0: 5, 1: 15} 2022-03-17T21:28:34.9880185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.9881644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.0385127Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.0386314Z 5 with iteration mapping 2022-03-17T21:28:35.0387145Z {0: 0, 1: 2} 2022-03-17T21:28:35.0486272Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.0487289Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.0488574Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.0489986Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.0526947Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.0528060Z 5 with iteration mapping 2022-03-17T21:28:35.0528885Z {0: 0, 1: 3} 2022-03-17T21:28:35.0637067Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.0639500Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.0642694Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.0644792Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.0675712Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.0676837Z 5 with iteration mapping 2022-03-17T21:28:35.0677644Z {0: 0, 1: 10} 2022-03-17T21:28:35.0886779Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.0889084Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.0892101Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.0894234Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.0926705Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.0927882Z 5 with iteration mapping 2022-03-17T21:28:35.0928537Z {0: 5, 1: 7} 2022-03-17T21:28:35.1095535Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1097851Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1100850Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1102942Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1135692Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.1136805Z 5 with iteration mapping 2022-03-17T21:28:35.1137617Z {0: 5, 1: 8} 2022-03-17T21:28:35.1318676Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1321390Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1324580Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1326661Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1355584Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.1356670Z 5 with iteration mapping 2022-03-17T21:28:35.1357417Z {0: 5, 1: 15} 2022-03-17T21:28:35.1628672Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1630924Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1633832Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1635793Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1671888Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.1672956Z 6 with iteration mapping 2022-03-17T21:28:35.1673707Z {0: 0, 1: 2} 2022-03-17T21:28:35.1768551Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1770646Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1773464Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1775469Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1808506Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.1809370Z 6 with iteration mapping 2022-03-17T21:28:35.1810005Z {0: 0, 1: 3} 2022-03-17T21:28:35.1914133Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1916246Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1919097Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.1921451Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.1953257Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.1954192Z 6 with iteration mapping 2022-03-17T21:28:35.1954856Z {0: 0, 1: 10} 2022-03-17T21:28:35.2237124Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2239235Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2242369Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2244387Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2272792Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.2273679Z 6 with iteration mapping 2022-03-17T21:28:35.2274303Z {0: 5, 1: 7} 2022-03-17T21:28:35.2436298Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2438411Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2442141Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2444161Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2472411Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.2473278Z 6 with iteration mapping 2022-03-17T21:28:35.2473914Z {0: 5, 1: 8} 2022-03-17T21:28:35.2650806Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2652858Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2655692Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2657645Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2692276Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:35.2693209Z 6 with iteration mapping 2022-03-17T21:28:35.2694201Z {0: 5, 1: 15} 2022-03-17T21:28:35.2932500Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2934584Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2937485Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:35.2939448Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:35.2973601Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:35.2974442Z 7 with iteration mapping 2022-03-17T21:28:35.2975052Z {0: 0, 1: 2} 2022-03-17T21:28:35.3102419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.3288821Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:35.3289908Z 7 with iteration mapping 2022-03-17T21:28:35.3290783Z {0: 0, 1: 3} 2022-03-17T21:28:35.3468866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.3686395Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:35.3687188Z 7 with iteration mapping 2022-03-17T21:28:35.3687746Z {0: 0, 1: 10} 2022-03-17T21:28:35.3805858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.4131187Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:35.4132027Z 7 with iteration mapping 2022-03-17T21:28:35.4132637Z {0: 5, 1: 7} 2022-03-17T21:28:35.4242126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.4243339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.4521581Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:35.4522938Z 7 with iteration mapping 2022-03-17T21:28:35.4523577Z {0: 5, 1: 8} 2022-03-17T21:28:35.4632306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.4633468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.4924952Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:35.4926087Z 7 with iteration mapping 2022-03-17T21:28:35.4926942Z {0: 5, 1: 15} 2022-03-17T21:28:35.5054210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.5055313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.5495537Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:35.5496080Z 8 with iteration mapping 2022-03-17T21:28:35.5496395Z {0: 0, 1: 2} 2022-03-17T21:28:35.5609084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.5776525Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:35.5776953Z 8 with iteration mapping 2022-03-17T21:28:35.5777261Z {0: 0, 1: 3} 2022-03-17T21:28:35.5894635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.6085273Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:35.6085943Z 8 with iteration mapping 2022-03-17T21:28:35.6086371Z {0: 0, 1: 10} 2022-03-17T21:28:35.6205519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.6507692Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:35.6508447Z 8 with iteration mapping 2022-03-17T21:28:35.6508905Z {0: 5, 1: 7} 2022-03-17T21:28:35.6638300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.6639633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.6871713Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:35.6872634Z 8 with iteration mapping 2022-03-17T21:28:35.6873334Z {0: 5, 1: 8} 2022-03-17T21:28:35.6999441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.7000793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.7236461Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:35.7237373Z 8 with iteration mapping 2022-03-17T21:28:35.7238013Z {0: 5, 1: 15} 2022-03-17T21:28:35.7351616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.7352981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.7748457Z Running test: batch_norm_net sync interval 2022-03-17T21:28:35.7749132Z 1 with iteration mapping 2022-03-17T21:28:35.7749602Z {0: 0, 1: 2} 2022-03-17T21:28:35.7804931Z Running test: batch_norm_net sync interval 2022-03-17T21:28:35.7805541Z 1 with iteration mapping 2022-03-17T21:28:35.7805980Z {0: 0, 1: 3} 2022-03-17T21:28:35.7857657Z Running test: batch_norm_net sync interval 2022-03-17T21:28:35.7858273Z 1 with iteration mapping 2022-03-17T21:28:35.7858795Z {0: 0, 1: 10} 2022-03-17T21:28:35.7906533Z Running test: batch_norm_net sync interval 2022-03-17T21:28:35.7907483Z 1 with iteration mapping 2022-03-17T21:28:35.7908901Z {0: 5, 1: 7} 2022-03-17T21:28:35.8010439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.8011747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.8209728Z Running test: batch_norm_net sync interval 2022-03-17T21:28:35.8210559Z 1 with iteration mapping 2022-03-17T21:28:35.8211236Z {0: 5, 1: 8} 2022-03-17T21:28:35.8306005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.8307405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.8495325Z Running test: batch_norm_net sync interval 2022-03-17T21:28:35.8496224Z 1 with iteration mapping 2022-03-17T21:28:35.8496960Z {0: 5, 1: 15} 2022-03-17T21:28:35.8596465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.8597772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:35.8789026Z Running test: large_conv_model sync interval 2022-03-17T21:28:35.8789931Z 1 with iteration mapping 2022-03-17T21:28:35.8790627Z {0: 0, 1: 2} 2022-03-17T21:28:35.8846493Z Running test: large_conv_model sync interval 2022-03-17T21:28:35.8847256Z 1 with iteration mapping 2022-03-17T21:28:35.8848409Z {0: 0, 1: 3} 2022-03-17T21:28:35.8906708Z Running test: large_conv_model sync interval 2022-03-17T21:28:35.8907571Z 1 with iteration mapping 2022-03-17T21:28:35.8908297Z {0: 0, 1: 10} 2022-03-17T21:28:35.8961315Z Running test: large_conv_model sync interval 2022-03-17T21:28:35.8962379Z 1 with iteration mapping 2022-03-17T21:28:35.8963121Z {0: 5, 1: 7} 2022-03-17T21:28:36.0982423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:36.0983684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:36.8410011Z Running test: large_conv_model sync interval 2022-03-17T21:28:36.8410473Z 1 with iteration mapping 2022-03-17T21:28:36.8410814Z {0: 5, 1: 8} 2022-03-17T21:28:37.0358893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:37.0360397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:37.7783015Z Running test: large_conv_model sync interval 2022-03-17T21:28:37.7784043Z 1 with iteration mapping 2022-03-17T21:28:37.7784794Z {0: 5, 1: 15} 2022-03-17T21:28:37.9850328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:37.9851834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.7400425Z Running test: small_model sync interval 2022-03-17T21:28:38.7401396Z 1 with iteration mapping 2022-03-17T21:28:38.7402466Z {0: 0, 1: 2} 2022-03-17T21:28:38.7464824Z Running test: small_model sync interval 2022-03-17T21:28:38.7465612Z 1 with iteration mapping 2022-03-17T21:28:38.7466234Z {0: 0, 1: 3} 2022-03-17T21:28:38.7527764Z Running test: small_model sync interval 2022-03-17T21:28:38.7528554Z 1 with iteration mapping 2022-03-17T21:28:38.7529299Z {0: 0, 1: 10} 2022-03-17T21:28:38.7590828Z Running test: small_model sync interval 2022-03-17T21:28:38.7591942Z 1 with iteration mapping 2022-03-17T21:28:38.7592575Z {0: 5, 1: 7} 2022-03-17T21:28:38.7731985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.7733230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.8000005Z Running test: small_model sync interval 2022-03-17T21:28:38.8001108Z 1 with iteration mapping 2022-03-17T21:28:38.8001747Z {0: 5, 1: 8} 2022-03-17T21:28:38.8140525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.8141758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.8392541Z Running test: small_model sync interval 2022-03-17T21:28:38.8393670Z 1 with iteration mapping 2022-03-17T21:28:38.8394535Z {0: 5, 1: 15} 2022-03-17T21:28:38.8525661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.8527190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:38.8782961Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.8784048Z 1 with iteration mapping 2022-03-17T21:28:38.8784792Z {0: 0, 1: 2} 2022-03-17T21:28:38.8831379Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.8832531Z 1 with iteration mapping 2022-03-17T21:28:38.8833698Z {0: 0, 1: 3} 2022-03-17T21:28:38.8872135Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.8873187Z 1 with iteration mapping 2022-03-17T21:28:38.8873958Z {0: 0, 1: 10} 2022-03-17T21:28:38.8912340Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.8913403Z 1 with iteration mapping 2022-03-17T21:28:38.8914204Z {0: 5, 1: 7} 2022-03-17T21:28:38.9097221Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9098339Z 1 with iteration mapping 2022-03-17T21:28:38.9099097Z {0: 5, 1: 8} 2022-03-17T21:28:38.9277049Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9278130Z 1 with iteration mapping 2022-03-17T21:28:38.9278878Z {0: 5, 1: 15} 2022-03-17T21:28:38.9454265Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9455335Z 1 with iteration mapping 2022-03-17T21:28:38.9456082Z {0: 0, 1: 2} 2022-03-17T21:28:38.9502392Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9503452Z 1 with iteration mapping 2022-03-17T21:28:38.9504186Z {0: 0, 1: 3} 2022-03-17T21:28:38.9545779Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9546819Z 1 with iteration mapping 2022-03-17T21:28:38.9547585Z {0: 0, 1: 10} 2022-03-17T21:28:38.9587307Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9588336Z 1 with iteration mapping 2022-03-17T21:28:38.9589105Z {0: 5, 1: 7} 2022-03-17T21:28:38.9768601Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9769723Z 1 with iteration mapping 2022-03-17T21:28:38.9770493Z {0: 5, 1: 8} 2022-03-17T21:28:38.9946571Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:38.9947619Z 1 with iteration mapping 2022-03-17T21:28:38.9948358Z {0: 5, 1: 15} 2022-03-17T21:28:39.0125665Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:39.0127019Z 1 with iteration mapping 2022-03-17T21:28:39.0127781Z {0: 0, 1: 2} 2022-03-17T21:28:39.0188478Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:39.0189465Z 1 with iteration mapping 2022-03-17T21:28:39.0190238Z {0: 0, 1: 3} 2022-03-17T21:28:39.0248029Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:39.0249026Z 1 with iteration mapping 2022-03-17T21:28:39.0249813Z {0: 0, 1: 10} 2022-03-17T21:28:39.0310310Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:39.0311346Z 1 with iteration mapping 2022-03-17T21:28:39.0312113Z {0: 5, 1: 7} 2022-03-17T21:28:39.0443362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.0444845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.0691886Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:39.0693024Z 1 with iteration mapping 2022-03-17T21:28:39.0693805Z {0: 5, 1: 8} 2022-03-17T21:28:39.0820594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.0822073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.1073770Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:39.1075184Z 1 with iteration mapping 2022-03-17T21:28:39.1075959Z {0: 5, 1: 15} 2022-03-17T21:28:39.1206859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.1208338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.1462814Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:39.1463862Z 1 with iteration mapping 2022-03-17T21:28:39.1464606Z {0: 0, 1: 2} 2022-03-17T21:28:39.1523400Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:39.1524438Z 1 with iteration mapping 2022-03-17T21:28:39.1525203Z {0: 0, 1: 3} 2022-03-17T21:28:39.1584592Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:39.1585605Z 1 with iteration mapping 2022-03-17T21:28:39.1586412Z {0: 0, 1: 10} 2022-03-17T21:28:39.1648881Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:39.1649875Z 1 with iteration mapping 2022-03-17T21:28:39.1650644Z {0: 5, 1: 7} 2022-03-17T21:28:39.1779634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.1781161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.2035662Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:39.2036667Z 1 with iteration mapping 2022-03-17T21:28:39.2037457Z {0: 5, 1: 8} 2022-03-17T21:28:39.2166069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.2167554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.2483758Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:39.2484809Z 1 with iteration mapping 2022-03-17T21:28:39.2485558Z {0: 5, 1: 15} 2022-03-17T21:28:39.2617078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:39.2618576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:42.2044501Z ok (47.906s) 2022-03-17T21:28:42.2044978Z 2022-03-17T21:28:42.2046650Z ---------------------------------------------------------------------- 2022-03-17T21:28:42.2047643Z Ran 1 test in 47.907s 2022-03-17T21:28:42.2048092Z 2022-03-17T21:28:42.2048349Z OK 2022-03-17T21:28:42.2048710Z 2022-03-17T21:28:42.2048982Z Generating XML reports... 2022-03-17T21:28:42.2126258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212754.xml 2022-03-17T21:28:44.8077666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:28:44.8088113Z 2022-03-17T21:28:44.8088775Z Running tests... 2022-03-17T21:28:44.8090198Z ---------------------------------------------------------------------- 2022-03-17T21:28:45.4137451Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55687 2022-03-17T21:28:45.4174450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55688 2022-03-17T21:28:45.4176612Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:28:46.5510062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:28:46.5511251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:28:46.5513263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:46.5515522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:46.5617996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:28:46.6525099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:28:56.2202798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:56.2204091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:56.2530214Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:56.2532258Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:56.2535095Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:56.2537034Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:56.3389592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:56.3390829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:56.3535558Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:56.3537632Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:58.9438923Z ok (14.134s) 2022-03-17T21:28:58.9439295Z 2022-03-17T21:28:58.9440547Z ---------------------------------------------------------------------- 2022-03-17T21:28:58.9442011Z Ran 1 test in 14.135s 2022-03-17T21:28:58.9443289Z 2022-03-17T21:28:58.9443974Z OK 2022-03-17T21:28:58.9444318Z 2022-03-17T21:28:58.9444607Z Generating XML reports... 2022-03-17T21:28:58.9519168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212844.xml 2022-03-17T21:29:01.5477311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:01.5489119Z 2022-03-17T21:29:01.5489550Z Running tests... 2022-03-17T21:29:01.5490009Z ---------------------------------------------------------------------- 2022-03-17T21:29:02.1695463Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55901 2022-03-17T21:29:02.1746205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55902 2022-03-17T21:29:02.1748328Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:29:03.2771507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:03.2772767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:03.2774822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:03.2777136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:03.2879574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:03.3786272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:15.4042802Z ok (13.854s) 2022-03-17T21:29:15.4043419Z 2022-03-17T21:29:15.4044892Z ---------------------------------------------------------------------- 2022-03-17T21:29:15.4046096Z Ran 1 test in 13.855s 2022-03-17T21:29:15.4046579Z 2022-03-17T21:29:15.4046837Z OK 2022-03-17T21:29:15.4047278Z 2022-03-17T21:29:15.4047723Z Generating XML reports... 2022-03-17T21:29:15.4110252Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212901.xml 2022-03-17T21:29:18.0804798Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:18.0815708Z 2022-03-17T21:29:18.0816246Z Running tests... 2022-03-17T21:29:18.0817410Z ---------------------------------------------------------------------- 2022-03-17T21:29:18.6820830Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56115 2022-03-17T21:29:18.6863271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56116 2022-03-17T21:29:18.6864625Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:29:19.8096798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:19.8099021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:19.8100335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:19.8102032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:19.8204588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:19.9117339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:19.9328530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:19.9330483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:19.9331797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:19.9333474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:21.7964852Z ok (3.715s) 2022-03-17T21:29:21.7965556Z 2022-03-17T21:29:21.7966869Z ---------------------------------------------------------------------- 2022-03-17T21:29:21.7967742Z Ran 1 test in 3.715s 2022-03-17T21:29:21.7968118Z 2022-03-17T21:29:21.7968307Z OK 2022-03-17T21:29:21.7968602Z 2022-03-17T21:29:21.7969693Z Generating XML reports... 2022-03-17T21:29:21.8046755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212918.xml 2022-03-17T21:29:24.7902942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:24.7913633Z 2022-03-17T21:29:24.7913934Z Running tests... 2022-03-17T21:29:24.7915168Z ---------------------------------------------------------------------- 2022-03-17T21:29:25.4046693Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56331 2022-03-17T21:29:25.4108451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56332 2022-03-17T21:29:25.4110356Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:29:26.5314097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:26.5315377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:26.5317435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:26.5416900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:26.5418260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:26.6336230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:26.6544741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:26.6546611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:26.6547901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:26.6549603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:28.5217550Z ok (3.730s) 2022-03-17T21:29:28.5217943Z 2022-03-17T21:29:28.5219372Z ---------------------------------------------------------------------- 2022-03-17T21:29:28.5220165Z Ran 1 test in 3.730s 2022-03-17T21:29:28.5220531Z 2022-03-17T21:29:28.5220721Z OK 2022-03-17T21:29:28.5221013Z 2022-03-17T21:29:28.5221292Z Generating XML reports... 2022-03-17T21:29:28.5299894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212924.xml 2022-03-17T21:29:31.5719588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:31.5730208Z 2022-03-17T21:29:31.5730718Z Running tests... 2022-03-17T21:29:31.5731690Z ---------------------------------------------------------------------- 2022-03-17T21:29:32.1756520Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56547 2022-03-17T21:29:32.1799160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56548 2022-03-17T21:29:32.1801042Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:29:33.3258150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:33.3258622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:33.3259459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:33.3260152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:33.3276768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:33.3277252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:42.8490922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:29:42.8492159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:29:42.8622364Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:29:42.8627082Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:29:45.5117835Z ok (13.938s) 2022-03-17T21:29:45.5118222Z 2022-03-17T21:29:45.5119239Z ---------------------------------------------------------------------- 2022-03-17T21:29:45.5120036Z Ran 1 test in 13.939s 2022-03-17T21:29:45.5120403Z 2022-03-17T21:29:45.5124758Z OK 2022-03-17T21:29:45.5125338Z 2022-03-17T21:29:45.5125691Z Generating XML reports... 2022-03-17T21:29:45.5187453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212931.xml 2022-03-17T21:29:48.4436476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:48.4450572Z 2022-03-17T21:29:48.4451055Z Running tests... 2022-03-17T21:29:48.4452084Z ---------------------------------------------------------------------- 2022-03-17T21:29:49.0756668Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56761 2022-03-17T21:29:49.0808557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56762 2022-03-17T21:29:49.0810006Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:29:50.2487005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:50.2488511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:50.2490523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:50.2492711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:50.2496829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:50.3500428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:59.7465163Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:30:00.0464440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:30:00.0470258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:30:02.7107015Z ok (14.265s) 2022-03-17T21:30:02.7107533Z 2022-03-17T21:30:02.7108813Z ---------------------------------------------------------------------- 2022-03-17T21:30:02.7109683Z Ran 1 test in 14.266s 2022-03-17T21:30:02.7110043Z 2022-03-17T21:30:02.7110226Z OK 2022-03-17T21:30:02.7110497Z 2022-03-17T21:30:02.7111404Z Generating XML reports... 2022-03-17T21:30:02.7186907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212948.xml 2022-03-17T21:30:05.3797423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:05.3812683Z 2022-03-17T21:30:05.3813306Z Running tests... 2022-03-17T21:30:05.3814688Z ---------------------------------------------------------------------- 2022-03-17T21:30:05.9905512Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56975 2022-03-17T21:30:05.9942889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56976 2022-03-17T21:30:05.9944458Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:07.2298710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:07.2300177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:07.2302184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:07.2401707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:07.2405869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:07.3322706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:09.2028877Z ok (3.821s) 2022-03-17T21:30:09.2029310Z 2022-03-17T21:30:09.2030344Z ---------------------------------------------------------------------- 2022-03-17T21:30:09.2031124Z Ran 1 test in 3.822s 2022-03-17T21:30:09.2031496Z 2022-03-17T21:30:09.2034833Z OK 2022-03-17T21:30:09.2035233Z 2022-03-17T21:30:09.2035543Z Generating XML reports... 2022-03-17T21:30:09.2105522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213005.xml 2022-03-17T21:30:11.8180645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:11.8196315Z 2022-03-17T21:30:11.8196580Z Running tests... 2022-03-17T21:30:11.8197166Z ---------------------------------------------------------------------- 2022-03-17T21:30:12.4252147Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57185 2022-03-17T21:30:12.4297645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57186 2022-03-17T21:30:12.4299218Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:13.5652299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:13.5654776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:13.5656214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:13.5657953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:13.5762938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:13.6664610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:15.4392754Z ok (3.619s) 2022-03-17T21:30:15.4393367Z 2022-03-17T21:30:15.4394453Z ---------------------------------------------------------------------- 2022-03-17T21:30:15.4395408Z Ran 1 test in 3.620s 2022-03-17T21:30:15.4395838Z 2022-03-17T21:30:15.4396024Z OK 2022-03-17T21:30:15.4396970Z 2022-03-17T21:30:15.4398296Z Generating XML reports... 2022-03-17T21:30:15.4466742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213011.xml 2022-03-17T21:30:18.3182950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:18.3194504Z 2022-03-17T21:30:18.3194788Z Running tests... 2022-03-17T21:30:18.3195338Z ---------------------------------------------------------------------- 2022-03-17T21:30:18.9329349Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57401 2022-03-17T21:30:18.9381606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57402 2022-03-17T21:30:18.9382916Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:20.0945495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:20.0946753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:20.0947355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:20.0948097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:20.0963103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:20.1955257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:21.9481457Z ok (3.628s) 2022-03-17T21:30:21.9482288Z 2022-03-17T21:30:21.9483167Z ---------------------------------------------------------------------- 2022-03-17T21:30:21.9484320Z Ran 1 test in 3.629s 2022-03-17T21:30:21.9484856Z 2022-03-17T21:30:21.9485068Z OK 2022-03-17T21:30:21.9485384Z 2022-03-17T21:30:21.9485662Z Generating XML reports... 2022-03-17T21:30:21.9567669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213018.xml 2022-03-17T21:30:25.0056922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:25.0066576Z 2022-03-17T21:30:25.0066858Z Running tests... 2022-03-17T21:30:25.0068376Z ---------------------------------------------------------------------- 2022-03-17T21:30:25.0082921Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:30:25.0083704Z 2022-03-17T21:30:25.0084526Z ---------------------------------------------------------------------- 2022-03-17T21:30:25.0085282Z Ran 1 test in 0.002s 2022-03-17T21:30:25.0085649Z 2022-03-17T21:30:25.0085875Z OK (skipped=1) 2022-03-17T21:30:25.0086217Z 2022-03-17T21:30:25.0086464Z Generating XML reports... 2022-03-17T21:30:25.0125024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213025.xml 2022-03-17T21:30:27.6684941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:27.6697260Z 2022-03-17T21:30:27.6697582Z Running tests... 2022-03-17T21:30:27.6698149Z ---------------------------------------------------------------------- 2022-03-17T21:30:28.2726276Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57678 2022-03-17T21:30:28.2761704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57679 2022-03-17T21:30:28.2763393Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:29.4290977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:29.4292482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:29.4293825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:29.4295142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:29.4399545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:29.5313254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:29.5524039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:30:29.5525863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:29.5527157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:30:29.5528823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:31.4864925Z ok (3.816s) 2022-03-17T21:30:31.4865315Z 2022-03-17T21:30:31.4866146Z ---------------------------------------------------------------------- 2022-03-17T21:30:31.4866908Z Ran 1 test in 3.817s 2022-03-17T21:30:31.4867286Z 2022-03-17T21:30:31.4867472Z OK 2022-03-17T21:30:31.4867767Z 2022-03-17T21:30:31.4868026Z Generating XML reports... 2022-03-17T21:30:31.4944794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213027.xml 2022-03-17T21:30:34.1573645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:34.1583374Z 2022-03-17T21:30:34.1583819Z Running tests... 2022-03-17T21:30:34.1585227Z ---------------------------------------------------------------------- 2022-03-17T21:30:34.7751819Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57900 2022-03-17T21:30:34.7789841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57901 2022-03-17T21:30:34.7791458Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:35.9367490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:35.9368440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:35.9369515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:35.9472905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:35.9474813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:36.0388202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:38.3886395Z skip (4.230s) 2022-03-17T21:30:38.3887028Z 2022-03-17T21:30:38.3888119Z ---------------------------------------------------------------------- 2022-03-17T21:30:38.3889192Z Ran 1 test in 4.230s 2022-03-17T21:30:38.3889693Z 2022-03-17T21:30:38.3889990Z OK (skipped=1) 2022-03-17T21:30:38.3890357Z 2022-03-17T21:30:38.3890679Z Generating XML reports... 2022-03-17T21:30:38.3975653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213034.xml 2022-03-17T21:30:41.3890521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:41.3901914Z 2022-03-17T21:30:41.3902122Z Running tests... 2022-03-17T21:30:41.3902817Z ---------------------------------------------------------------------- 2022-03-17T21:30:42.0170293Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58110 2022-03-17T21:30:42.0218686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58111 2022-03-17T21:30:42.0220013Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:43.1990964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:43.1992298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:43.1994386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:43.1996190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:43.1997509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:43.1998696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:44.9310169Z ok (3.540s) 2022-03-17T21:30:44.9310621Z 2022-03-17T21:30:44.9311731Z ---------------------------------------------------------------------- 2022-03-17T21:30:44.9312649Z Ran 1 test in 3.541s 2022-03-17T21:30:44.9313100Z 2022-03-17T21:30:44.9313345Z OK 2022-03-17T21:30:44.9313709Z 2022-03-17T21:30:44.9314108Z Generating XML reports... 2022-03-17T21:30:44.9388133Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213041.xml 2022-03-17T21:30:47.6686787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:47.6695209Z 2022-03-17T21:30:47.6695622Z Running tests... 2022-03-17T21:30:47.6696977Z ---------------------------------------------------------------------- 2022-03-17T21:30:48.2731202Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58320 2022-03-17T21:30:48.2769132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58321 2022-03-17T21:30:48.2770714Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:49.3658145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:49.3659690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:49.3661893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:49.3760842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:49.3775326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:49.4771898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:49.5185662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:30:49.5187198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:30:49.5189257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:49.5287237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:49.5553832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:30:49.5555391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:30:49.5557773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:30:49.5559577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:30:49.5690952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T21:30:49.5692497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T21:30:49.5694543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:30:49.5795676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:30:51.3859944Z ok (3.716s) 2022-03-17T21:30:51.3860575Z 2022-03-17T21:30:51.3861572Z ---------------------------------------------------------------------- 2022-03-17T21:30:51.3862555Z Ran 1 test in 3.716s 2022-03-17T21:30:51.3862978Z 2022-03-17T21:30:51.3863236Z OK 2022-03-17T21:30:51.3864231Z 2022-03-17T21:30:51.3864686Z Generating XML reports... 2022-03-17T21:30:51.3942644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213047.xml 2022-03-17T21:30:54.2908473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:54.2917921Z 2022-03-17T21:30:54.2918343Z Running tests... 2022-03-17T21:30:54.2919444Z ---------------------------------------------------------------------- 2022-03-17T21:30:54.9241670Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58554 2022-03-17T21:30:54.9289199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58555 2022-03-17T21:30:54.9290538Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:30:56.0849099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:56.0850347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:56.0852399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:56.0951902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:56.0953535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:56.1871591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:56.2081382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:30:56.2082823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:30:56.2084610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:56.2086383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:57.9423557Z ok (3.650s) 2022-03-17T21:30:57.9423928Z 2022-03-17T21:30:57.9424829Z ---------------------------------------------------------------------- 2022-03-17T21:30:57.9425650Z Ran 1 test in 3.650s 2022-03-17T21:30:57.9426024Z 2022-03-17T21:30:57.9426206Z OK 2022-03-17T21:30:57.9426502Z 2022-03-17T21:30:57.9426794Z Generating XML reports... 2022-03-17T21:30:57.9502591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213054.xml 2022-03-17T21:31:00.5879857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:00.5887412Z 2022-03-17T21:31:00.5887950Z Running tests... 2022-03-17T21:31:00.5889147Z ---------------------------------------------------------------------- 2022-03-17T21:31:01.1965936Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58770 2022-03-17T21:31:01.2005226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58771 2022-03-17T21:31:01.2006616Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:02.3032749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:02.3034048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:02.3036086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:02.3037879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:02.3142434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:02.4044006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:04.1097912Z ok (3.520s) 2022-03-17T21:31:04.1098720Z 2022-03-17T21:31:04.1100143Z ---------------------------------------------------------------------- 2022-03-17T21:31:04.1101140Z Ran 1 test in 3.521s 2022-03-17T21:31:04.1101540Z 2022-03-17T21:31:04.1101731Z OK 2022-03-17T21:31:04.1102020Z 2022-03-17T21:31:04.1102301Z Generating XML reports... 2022-03-17T21:31:04.1183946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213100.xml 2022-03-17T21:31:06.8950330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:06.8960876Z 2022-03-17T21:31:06.8961153Z Running tests... 2022-03-17T21:31:06.8961588Z ---------------------------------------------------------------------- 2022-03-17T21:31:07.5004024Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58980 2022-03-17T21:31:07.5044343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58981 2022-03-17T21:31:07.5046260Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:08.6559365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:08.6562657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:08.6564662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:08.6567414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:08.6669005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:08.7582225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:10.7137078Z ok (3.817s) 2022-03-17T21:31:10.7137598Z 2022-03-17T21:31:10.7138627Z ---------------------------------------------------------------------- 2022-03-17T21:31:10.7139401Z Ran 1 test in 3.817s 2022-03-17T21:31:10.7139766Z 2022-03-17T21:31:10.7139951Z OK 2022-03-17T21:31:10.7140224Z 2022-03-17T21:31:10.7140504Z Generating XML reports... 2022-03-17T21:31:10.7220374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213106.xml 2022-03-17T21:31:13.3274266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:13.3289022Z 2022-03-17T21:31:13.3289450Z Running tests... 2022-03-17T21:31:13.3289980Z ---------------------------------------------------------------------- 2022-03-17T21:31:13.9583559Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59190 2022-03-17T21:31:13.9619278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59191 2022-03-17T21:31:13.9620559Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:15.1062165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:15.1063110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:15.1064891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:15.1067041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:15.1077029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:15.2076716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:15.2197504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:31:15.2199415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:15.2200710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:31:15.2202522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:17.0724146Z ok (3.743s) 2022-03-17T21:31:17.0724541Z 2022-03-17T21:31:17.0726618Z ---------------------------------------------------------------------- 2022-03-17T21:31:17.0727520Z Ran 1 test in 3.743s 2022-03-17T21:31:17.0727887Z 2022-03-17T21:31:17.0728090Z OK 2022-03-17T21:31:17.0728409Z 2022-03-17T21:31:17.0728703Z Generating XML reports... 2022-03-17T21:31:17.0810217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213113.xml 2022-03-17T21:31:19.6745884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:19.6760670Z 2022-03-17T21:31:19.6761254Z Running tests... 2022-03-17T21:31:19.6762512Z ---------------------------------------------------------------------- 2022-03-17T21:31:20.3083978Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59406 2022-03-17T21:31:20.3123320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59407 2022-03-17T21:31:20.3124619Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:21.4453506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:21.4454764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:21.4456822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:21.4553116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:21.4560442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:21.5563992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:21.5772289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:31:21.5773063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:31:21.5773919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:21.5873390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:23.9226917Z ok (4.246s) 2022-03-17T21:31:23.9227388Z 2022-03-17T21:31:23.9228446Z ---------------------------------------------------------------------- 2022-03-17T21:31:23.9229229Z Ran 1 test in 4.246s 2022-03-17T21:31:23.9229595Z 2022-03-17T21:31:23.9229789Z OK 2022-03-17T21:31:23.9230508Z 2022-03-17T21:31:23.9231156Z Generating XML reports... 2022-03-17T21:31:23.9304242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213119.xml 2022-03-17T21:31:26.5270648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:26.5278871Z 2022-03-17T21:31:26.5279139Z Running tests... 2022-03-17T21:31:26.5279693Z ---------------------------------------------------------------------- 2022-03-17T21:31:27.1284040Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59622 2022-03-17T21:31:27.1322028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59623 2022-03-17T21:31:27.1323367Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:28.2858384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:28.2874628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:28.2876622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:28.2961600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:28.2968348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:28.3889117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:40.8576861Z ok (14.329s) 2022-03-17T21:31:40.8577275Z 2022-03-17T21:31:40.8578330Z ---------------------------------------------------------------------- 2022-03-17T21:31:40.8579090Z Ran 1 test in 14.330s 2022-03-17T21:31:40.8579451Z 2022-03-17T21:31:40.8579654Z OK 2022-03-17T21:31:40.8579943Z 2022-03-17T21:31:40.8580201Z Generating XML reports... 2022-03-17T21:31:40.8655845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213126.xml 2022-03-17T21:31:43.5370860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:43.5383500Z 2022-03-17T21:31:43.5384009Z Running tests... 2022-03-17T21:31:43.5385149Z ---------------------------------------------------------------------- 2022-03-17T21:31:44.1662410Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59836 2022-03-17T21:31:44.1706838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59837 2022-03-17T21:31:44.1708163Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:45.3179233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:45.3180500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:45.3182899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:45.3283089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:45.3285083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:45.4188856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:47.1805066Z ok (3.641s) 2022-03-17T21:31:47.1805486Z 2022-03-17T21:31:47.1806461Z ---------------------------------------------------------------------- 2022-03-17T21:31:47.1807424Z Ran 1 test in 3.642s 2022-03-17T21:31:47.1807803Z 2022-03-17T21:31:47.1808003Z OK 2022-03-17T21:31:47.1808294Z 2022-03-17T21:31:47.1808580Z Generating XML reports... 2022-03-17T21:31:47.1892928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213143.xml 2022-03-17T21:31:49.7504919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:49.7517754Z 2022-03-17T21:31:49.7518081Z Running tests... 2022-03-17T21:31:49.7519262Z ---------------------------------------------------------------------- 2022-03-17T21:31:50.3555582Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60046 2022-03-17T21:31:50.3590472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60047 2022-03-17T21:31:50.3591824Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:51.5095618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:51.5096885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:51.5098765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:51.5200823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:51.5213383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:51.6215021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:53.3688891Z ok (3.617s) 2022-03-17T21:31:53.3689262Z 2022-03-17T21:31:53.3690120Z ---------------------------------------------------------------------- 2022-03-17T21:31:53.3690899Z Ran 1 test in 3.617s 2022-03-17T21:31:53.3691268Z 2022-03-17T21:31:53.3692602Z OK 2022-03-17T21:31:53.3692978Z 2022-03-17T21:31:53.3693262Z Generating XML reports... 2022-03-17T21:31:53.3771457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213149.xml 2022-03-17T21:31:55.9861052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:55.9870542Z 2022-03-17T21:31:55.9870999Z Running tests... 2022-03-17T21:31:55.9872413Z ---------------------------------------------------------------------- 2022-03-17T21:31:56.6092658Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60256 2022-03-17T21:31:56.6148414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60257 2022-03-17T21:31:56.6149726Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:31:57.7837412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:57.7838358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:57.7839445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:57.7939143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:57.7958232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:57.8953677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:00.5245028Z ok (4.537s) 2022-03-17T21:32:00.5245585Z 2022-03-17T21:32:00.5246847Z ---------------------------------------------------------------------- 2022-03-17T21:32:00.5247763Z Ran 1 test in 4.537s 2022-03-17T21:32:00.5248152Z 2022-03-17T21:32:00.5248331Z OK 2022-03-17T21:32:00.5248652Z 2022-03-17T21:32:00.5251004Z Generating XML reports... 2022-03-17T21:32:00.5327757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213155.xml 2022-03-17T21:32:03.1783497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:03.1794930Z 2022-03-17T21:32:03.1795256Z Running tests... 2022-03-17T21:32:03.1796642Z ---------------------------------------------------------------------- 2022-03-17T21:32:03.7794833Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60472 2022-03-17T21:32:03.7831732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60473 2022-03-17T21:32:03.7833088Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:04.9429585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:04.9431525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:04.9432898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:04.9434624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:04.9448763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:05.0442742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:06.8956193Z ok (3.716s) 2022-03-17T21:32:06.8956700Z 2022-03-17T21:32:06.8957540Z ---------------------------------------------------------------------- 2022-03-17T21:32:06.8958307Z Ran 1 test in 3.716s 2022-03-17T21:32:06.8958691Z 2022-03-17T21:32:06.8958873Z OK 2022-03-17T21:32:06.8959147Z 2022-03-17T21:32:06.8959421Z Generating XML reports... 2022-03-17T21:32:06.9025511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213203.xml 2022-03-17T21:32:09.5295574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:09.5306920Z 2022-03-17T21:32:09.5307444Z Running tests... 2022-03-17T21:32:09.5308774Z ---------------------------------------------------------------------- 2022-03-17T21:32:10.1280133Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60688 2022-03-17T21:32:10.1322463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60689 2022-03-17T21:32:10.1323985Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:11.2870091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:11.2872650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:11.2873951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:11.2875591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:11.2884951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:11.3880600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:13.0413519Z skip (3.510s) 2022-03-17T21:32:13.0414329Z 2022-03-17T21:32:13.0416413Z ---------------------------------------------------------------------- 2022-03-17T21:32:13.0417304Z Ran 1 test in 3.510s 2022-03-17T21:32:13.0417675Z 2022-03-17T21:32:13.0417901Z OK (skipped=1) 2022-03-17T21:32:13.0418316Z 2022-03-17T21:32:13.0418587Z Generating XML reports... 2022-03-17T21:32:13.0497076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213209.xml 2022-03-17T21:32:15.5892970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:15.5903323Z 2022-03-17T21:32:15.5904016Z Running tests... 2022-03-17T21:32:15.5905045Z ---------------------------------------------------------------------- 2022-03-17T21:32:16.1886541Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60898 2022-03-17T21:32:16.1928722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60899 2022-03-17T21:32:16.1930068Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:17.3619973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:17.3621270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:17.3623194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:17.3625335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:17.3628710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:17.3629842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:19.6016390Z skip (4.011s) 2022-03-17T21:32:19.6016805Z 2022-03-17T21:32:19.6017807Z ---------------------------------------------------------------------- 2022-03-17T21:32:19.6018564Z Ran 1 test in 4.011s 2022-03-17T21:32:19.6018926Z 2022-03-17T21:32:19.6019189Z OK (skipped=1) 2022-03-17T21:32:19.6019530Z 2022-03-17T21:32:19.6019799Z Generating XML reports... 2022-03-17T21:32:19.6097744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213215.xml 2022-03-17T21:32:22.1602923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:22.1611775Z 2022-03-17T21:32:22.1612047Z Running tests... 2022-03-17T21:32:22.1613183Z ---------------------------------------------------------------------- 2022-03-17T21:32:22.7606759Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61108 2022-03-17T21:32:22.7647322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61109 2022-03-17T21:32:22.7648619Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:23.9185162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:23.9187017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:23.9188302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:23.9189941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:23.9293423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:24.0204930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:25.6741464Z skip (3.512s) 2022-03-17T21:32:25.6742008Z 2022-03-17T21:32:25.6743133Z ---------------------------------------------------------------------- 2022-03-17T21:32:25.6744347Z Ran 1 test in 3.513s 2022-03-17T21:32:25.6744726Z 2022-03-17T21:32:25.6745059Z OK (skipped=1) 2022-03-17T21:32:25.6745586Z 2022-03-17T21:32:25.6745873Z Generating XML reports... 2022-03-17T21:32:25.6824353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213222.xml 2022-03-17T21:32:28.2687955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:28.2696653Z 2022-03-17T21:32:28.2697141Z Running tests... 2022-03-17T21:32:28.2698265Z ---------------------------------------------------------------------- 2022-03-17T21:32:28.8753204Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61318 2022-03-17T21:32:28.8788547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61319 2022-03-17T21:32:28.8790147Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:30.0081064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:30.0083554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:30.0084908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:30.0086896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:30.0091350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:30.1091264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:32.1324359Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-03-17T21:32:33.9917866Z ok (5.721s) 2022-03-17T21:32:33.9918475Z 2022-03-17T21:32:33.9919535Z ---------------------------------------------------------------------- 2022-03-17T21:32:33.9920489Z Ran 1 test in 5.722s 2022-03-17T21:32:33.9920928Z 2022-03-17T21:32:33.9921114Z OK 2022-03-17T21:32:33.9921408Z 2022-03-17T21:32:33.9921683Z Generating XML reports... 2022-03-17T21:32:33.9999186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213228.xml 2022-03-17T21:32:36.6353514Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:36.6364586Z 2022-03-17T21:32:36.6365171Z Running tests... 2022-03-17T21:32:36.6366572Z ---------------------------------------------------------------------- 2022-03-17T21:32:37.2386447Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61528 2022-03-17T21:32:37.2418792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61529 2022-03-17T21:32:37.2420782Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:38.3877691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:38.3878898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:38.3879555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:38.3880371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:38.3997486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:38.4894504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:38.5009710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:32:38.5012397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:38.5014400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:32:38.5017156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:38.5019158Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-03-17T21:32:38.5020734Z No ranks successfully processed in monitoredBarrier. 2022-03-17T21:32:38.5117923Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-03-17T21:32:40.3499811Z ok (3.713s) 2022-03-17T21:32:40.3500251Z 2022-03-17T21:32:40.3501629Z ---------------------------------------------------------------------- 2022-03-17T21:32:40.3502456Z Ran 1 test in 3.713s 2022-03-17T21:32:40.3502834Z 2022-03-17T21:32:40.3503022Z OK 2022-03-17T21:32:40.3503318Z 2022-03-17T21:32:40.3503598Z Generating XML reports... 2022-03-17T21:32:40.3580349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213236.xml 2022-03-17T21:32:42.9750837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:42.9759725Z 2022-03-17T21:32:42.9760811Z Running tests... 2022-03-17T21:32:42.9762550Z ---------------------------------------------------------------------- 2022-03-17T21:32:43.5757621Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61744 2022-03-17T21:32:43.5790977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61745 2022-03-17T21:32:43.5792564Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:44.7255941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:44.7257850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:44.7260812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:44.7263664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:44.7272056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:44.7273651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:44.7382225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:32:44.7383999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:32:44.7385528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:44.7386955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:44.8638455Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:32:46.5874439Z ok (3.611s) 2022-03-17T21:32:46.5875063Z 2022-03-17T21:32:46.5876427Z ---------------------------------------------------------------------- 2022-03-17T21:32:46.5877564Z Ran 1 test in 3.611s 2022-03-17T21:32:46.5878025Z 2022-03-17T21:32:46.5878241Z OK 2022-03-17T21:32:46.5878575Z 2022-03-17T21:32:46.5878891Z Generating XML reports... 2022-03-17T21:32:46.5948563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213242.xml 2022-03-17T21:32:49.1554264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:49.1566739Z 2022-03-17T21:32:49.1566989Z Running tests... 2022-03-17T21:32:49.1567520Z ---------------------------------------------------------------------- 2022-03-17T21:32:49.7538313Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61960 2022-03-17T21:32:49.7568619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61961 2022-03-17T21:32:49.7570335Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:32:50.9046626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:50.9047531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:50.9048148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:50.9048900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:50.9060218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:51.0060523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:52.6664868Z skip (3.509s) 2022-03-17T21:32:52.6665298Z 2022-03-17T21:32:52.6666290Z ---------------------------------------------------------------------- 2022-03-17T21:32:52.6667061Z Ran 1 test in 3.510s 2022-03-17T21:32:52.6667410Z 2022-03-17T21:32:52.6671866Z OK (skipped=1) 2022-03-17T21:32:52.6672286Z 2022-03-17T21:32:52.6672570Z Generating XML reports... 2022-03-17T21:32:52.6743782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213249.xml 2022-03-17T21:32:55.2792257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:55.2804071Z 2022-03-17T21:32:55.2804364Z Running tests... 2022-03-17T21:32:55.2804912Z ---------------------------------------------------------------------- 2022-03-17T21:32:55.2829611Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:32:55.2830046Z 2022-03-17T21:32:55.2830512Z ---------------------------------------------------------------------- 2022-03-17T21:32:55.2830964Z Ran 1 test in 0.003s 2022-03-17T21:32:55.2831184Z 2022-03-17T21:32:55.2831311Z OK (skipped=1) 2022-03-17T21:32:55.2831469Z 2022-03-17T21:32:55.2831596Z Generating XML reports... 2022-03-17T21:32:55.2868329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213255.xml 2022-03-17T21:32:57.7601029Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:57.7612708Z 2022-03-17T21:32:57.7612987Z Running tests... 2022-03-17T21:32:57.7613535Z ---------------------------------------------------------------------- 2022-03-17T21:32:57.7637023Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:32:57.7637374Z 2022-03-17T21:32:57.7637862Z ---------------------------------------------------------------------- 2022-03-17T21:32:57.7638243Z Ran 1 test in 0.003s 2022-03-17T21:32:57.7638417Z 2022-03-17T21:32:57.7638526Z OK (skipped=1) 2022-03-17T21:32:57.7638677Z 2022-03-17T21:32:57.7638803Z Generating XML reports... 2022-03-17T21:32:57.7677304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213257.xml 2022-03-17T21:33:00.3220136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:00.3229006Z 2022-03-17T21:33:00.3229479Z Running tests... 2022-03-17T21:33:00.3230035Z ---------------------------------------------------------------------- 2022-03-17T21:33:00.3250438Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:00.3250702Z 2022-03-17T21:33:00.3251172Z ---------------------------------------------------------------------- 2022-03-17T21:33:00.3251463Z Ran 1 test in 0.002s 2022-03-17T21:33:00.3251607Z 2022-03-17T21:33:00.3251700Z OK (skipped=1) 2022-03-17T21:33:00.3251884Z 2022-03-17T21:33:00.3252008Z Generating XML reports... 2022-03-17T21:33:00.3291203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213300.xml 2022-03-17T21:33:02.8259037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:02.8275498Z 2022-03-17T21:33:02.8275799Z Running tests... 2022-03-17T21:33:02.8276320Z ---------------------------------------------------------------------- 2022-03-17T21:33:02.8305884Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:33:02.8306159Z 2022-03-17T21:33:02.8306540Z ---------------------------------------------------------------------- 2022-03-17T21:33:02.8306887Z Ran 1 test in 0.003s 2022-03-17T21:33:02.8307062Z 2022-03-17T21:33:02.8307174Z OK (skipped=1) 2022-03-17T21:33:02.8307346Z 2022-03-17T21:33:02.8307462Z Generating XML reports... 2022-03-17T21:33:02.8351489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213302.xml 2022-03-17T21:33:05.4861451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:05.4873139Z 2022-03-17T21:33:05.4873508Z Running tests... 2022-03-17T21:33:05.4874083Z ---------------------------------------------------------------------- 2022-03-17T21:33:05.4895940Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:05.4896471Z 2022-03-17T21:33:05.4897053Z ---------------------------------------------------------------------- 2022-03-17T21:33:05.4897446Z Ran 1 test in 0.002s 2022-03-17T21:33:05.4897646Z 2022-03-17T21:33:05.4897792Z OK (skipped=1) 2022-03-17T21:33:05.4897989Z 2022-03-17T21:33:05.4898136Z Generating XML reports... 2022-03-17T21:33:05.4933234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213305.xml 2022-03-17T21:33:07.9589786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:07.9600741Z 2022-03-17T21:33:07.9601245Z Running tests... 2022-03-17T21:33:07.9601940Z ---------------------------------------------------------------------- 2022-03-17T21:33:07.9618802Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:07.9619035Z 2022-03-17T21:33:07.9620311Z ---------------------------------------------------------------------- 2022-03-17T21:33:07.9621594Z Ran 1 test in 0.002s 2022-03-17T21:33:07.9621981Z 2022-03-17T21:33:07.9622207Z OK (skipped=1) 2022-03-17T21:33:07.9622573Z 2022-03-17T21:33:07.9622850Z Generating XML reports... 2022-03-17T21:33:07.9658622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213307.xml 2022-03-17T21:33:10.5314040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:10.5325991Z 2022-03-17T21:33:10.5326329Z Running tests... 2022-03-17T21:33:10.5326863Z ---------------------------------------------------------------------- 2022-03-17T21:33:10.5350287Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:10.5350620Z 2022-03-17T21:33:10.5350996Z ---------------------------------------------------------------------- 2022-03-17T21:33:10.5351352Z Ran 1 test in 0.002s 2022-03-17T21:33:10.5351524Z 2022-03-17T21:33:10.5351636Z OK (skipped=1) 2022-03-17T21:33:10.5351798Z 2022-03-17T21:33:10.5351938Z Generating XML reports... 2022-03-17T21:33:10.5390201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213310.xml 2022-03-17T21:33:13.1646530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:13.1658608Z 2022-03-17T21:33:13.1659040Z Running tests... 2022-03-17T21:33:13.1660261Z ---------------------------------------------------------------------- 2022-03-17T21:33:13.1680396Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:13.1681232Z 2022-03-17T21:33:13.1682202Z ---------------------------------------------------------------------- 2022-03-17T21:33:13.1682996Z Ran 1 test in 0.002s 2022-03-17T21:33:13.1683363Z 2022-03-17T21:33:13.1683604Z OK (skipped=1) 2022-03-17T21:33:13.1683952Z 2022-03-17T21:33:13.1684253Z Generating XML reports... 2022-03-17T21:33:13.1717753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213313.xml 2022-03-17T21:33:15.6840520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:15.6850028Z 2022-03-17T21:33:15.6850345Z Running tests... 2022-03-17T21:33:15.6850923Z ---------------------------------------------------------------------- 2022-03-17T21:33:16.2843467Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62706 2022-03-17T21:33:16.2919720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62707 2022-03-17T21:33:16.2922265Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:33:17.4525173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:17.4525788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:17.4526771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:17.4527565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:17.4642692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:17.5539106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:19.1999713Z ok (3.514s) 2022-03-17T21:33:19.2000190Z 2022-03-17T21:33:19.2001346Z ---------------------------------------------------------------------- 2022-03-17T21:33:19.2002444Z Ran 1 test in 3.515s 2022-03-17T21:33:19.2002841Z 2022-03-17T21:33:19.2003603Z OK 2022-03-17T21:33:19.2003987Z 2022-03-17T21:33:19.2004325Z Generating XML reports... 2022-03-17T21:33:19.2082643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213315.xml 2022-03-17T21:33:21.7632352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:21.7648067Z 2022-03-17T21:33:21.7648330Z Running tests... 2022-03-17T21:33:21.7648800Z ---------------------------------------------------------------------- 2022-03-17T21:33:22.4001367Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62916 2022-03-17T21:33:22.4035841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62917 2022-03-17T21:33:22.4037167Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:33:23.5579953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:23.5580529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:23.5581580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:23.5683455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:23.5686017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:23.6597242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:25.3134256Z ok (3.548s) 2022-03-17T21:33:25.3134643Z 2022-03-17T21:33:25.3135657Z ---------------------------------------------------------------------- 2022-03-17T21:33:25.3136435Z Ran 1 test in 3.548s 2022-03-17T21:33:25.3136806Z 2022-03-17T21:33:25.3137011Z OK 2022-03-17T21:33:25.3137309Z 2022-03-17T21:33:25.3137591Z Generating XML reports... 2022-03-17T21:33:25.3217730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213321.xml 2022-03-17T21:33:27.9129370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:27.9140310Z 2022-03-17T21:33:27.9140691Z Running tests... 2022-03-17T21:33:27.9141442Z ---------------------------------------------------------------------- 2022-03-17T21:33:27.9158296Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:27.9158568Z 2022-03-17T21:33:27.9158892Z ---------------------------------------------------------------------- 2022-03-17T21:33:27.9159230Z Ran 1 test in 0.002s 2022-03-17T21:33:27.9159402Z 2022-03-17T21:33:27.9159519Z OK (skipped=1) 2022-03-17T21:33:27.9159679Z 2022-03-17T21:33:27.9159805Z Generating XML reports... 2022-03-17T21:33:27.9196594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213327.xml 2022-03-17T21:33:30.3997651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:30.4011903Z 2022-03-17T21:33:30.4012242Z Running tests... 2022-03-17T21:33:30.4012764Z ---------------------------------------------------------------------- 2022-03-17T21:33:30.4032077Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:30.4032432Z 2022-03-17T21:33:30.4032799Z ---------------------------------------------------------------------- 2022-03-17T21:33:30.4033148Z Ran 1 test in 0.002s 2022-03-17T21:33:30.4033314Z 2022-03-17T21:33:30.4033427Z OK (skipped=1) 2022-03-17T21:33:30.4033591Z 2022-03-17T21:33:30.4033703Z Generating XML reports... 2022-03-17T21:33:30.4076075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213330.xml 2022-03-17T21:33:33.0785574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:33.0800627Z 2022-03-17T21:33:33.0801158Z Running tests... 2022-03-17T21:33:33.0802508Z ---------------------------------------------------------------------- 2022-03-17T21:33:33.7111258Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63260 2022-03-17T21:33:33.7141928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63261 2022-03-17T21:33:33.7143251Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:33:34.8826427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:34.8827685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:34.8829706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:34.8930346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:34.8948190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:34.9938671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:47.2392477Z ok (14.159s) 2022-03-17T21:33:47.2393129Z 2022-03-17T21:33:47.2394374Z ---------------------------------------------------------------------- 2022-03-17T21:33:47.2395328Z Ran 1 test in 14.159s 2022-03-17T21:33:47.2395722Z 2022-03-17T21:33:47.2395964Z OK 2022-03-17T21:33:47.2396334Z 2022-03-17T21:33:47.2396672Z Generating XML reports... 2022-03-17T21:33:47.2462637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213333.xml 2022-03-17T21:33:49.8027990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:49.8043448Z 2022-03-17T21:33:49.8043962Z Running tests... 2022-03-17T21:33:49.8046282Z ---------------------------------------------------------------------- 2022-03-17T21:33:50.4129757Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63474 2022-03-17T21:33:50.4167411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63475 2022-03-17T21:33:50.4168771Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:33:51.5800398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:51.5801709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:51.5804007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:51.5805856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:51.5913243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:51.6812272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:04.0476642Z ok (14.243s) 2022-03-17T21:34:04.0477070Z 2022-03-17T21:34:04.0478017Z ---------------------------------------------------------------------- 2022-03-17T21:34:04.0478801Z Ran 1 test in 14.243s 2022-03-17T21:34:04.0479163Z 2022-03-17T21:34:04.0479356Z OK 2022-03-17T21:34:04.0479630Z 2022-03-17T21:34:04.0479918Z Generating XML reports... 2022-03-17T21:34:04.0524122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213349.xml 2022-03-17T21:34:06.6883946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:06.6895081Z 2022-03-17T21:34:06.6895942Z Running tests... 2022-03-17T21:34:06.6896759Z ---------------------------------------------------------------------- 2022-03-17T21:34:07.2911311Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63688 2022-03-17T21:34:07.2945049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63689 2022-03-17T21:34:07.2946403Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:08.4774591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:08.4775785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:08.4777692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:08.4779527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:08.4884077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:08.5791247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:11.1044337Z ok (4.414s) 2022-03-17T21:34:11.1044745Z 2022-03-17T21:34:11.1045600Z ---------------------------------------------------------------------- 2022-03-17T21:34:11.1046380Z Ran 1 test in 4.415s 2022-03-17T21:34:11.1046733Z 2022-03-17T21:34:11.1046915Z OK 2022-03-17T21:34:11.1047202Z 2022-03-17T21:34:11.1047498Z Generating XML reports... 2022-03-17T21:34:11.1127449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213406.xml 2022-03-17T21:34:13.7493167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:13.7502577Z 2022-03-17T21:34:13.7502928Z Running tests... 2022-03-17T21:34:13.7503471Z ---------------------------------------------------------------------- 2022-03-17T21:34:14.3507935Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63898 2022-03-17T21:34:14.3545703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63899 2022-03-17T21:34:14.3547591Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:15.4603955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:15.4604541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:15.4605489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:15.4606296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:15.4618450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:15.4619012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:25.3145070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:34:25.3148481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:34:25.3192771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:34:25.3194860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:34:28.1842621Z ok (14.433s) 2022-03-17T21:34:28.1843741Z 2022-03-17T21:34:28.1844910Z ---------------------------------------------------------------------- 2022-03-17T21:34:28.1845682Z Ran 1 test in 14.434s 2022-03-17T21:34:28.1846057Z 2022-03-17T21:34:28.1846263Z OK 2022-03-17T21:34:28.1846574Z 2022-03-17T21:34:28.1846885Z Generating XML reports... 2022-03-17T21:34:28.1922338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213413.xml 2022-03-17T21:34:30.8136289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:30.8150616Z 2022-03-17T21:34:30.8151066Z Running tests... 2022-03-17T21:34:30.8152195Z ---------------------------------------------------------------------- 2022-03-17T21:34:31.4488216Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64112 2022-03-17T21:34:31.4530451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64113 2022-03-17T21:34:31.4531772Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:32.5862900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:32.5863360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:32.5864125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:32.5864915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:32.5974315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:32.6885619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:32.7094309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:34:32.7094780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:34:32.7096162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:32.7096990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:35.1635657Z ok (4.348s) 2022-03-17T21:34:35.1636079Z 2022-03-17T21:34:35.1637046Z ---------------------------------------------------------------------- 2022-03-17T21:34:35.1637855Z Ran 1 test in 4.348s 2022-03-17T21:34:35.1638272Z 2022-03-17T21:34:35.1641648Z OK 2022-03-17T21:34:35.1642314Z 2022-03-17T21:34:35.1642606Z Generating XML reports... 2022-03-17T21:34:35.1706666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213430.xml 2022-03-17T21:34:37.8586016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:37.8597157Z 2022-03-17T21:34:37.8597603Z Running tests... 2022-03-17T21:34:37.8598060Z ---------------------------------------------------------------------- 2022-03-17T21:34:38.4558929Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64334 2022-03-17T21:34:38.4600761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64335 2022-03-17T21:34:38.4602034Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:39.5989861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:39.5992246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:39.5994930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:39.6089696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:39.6091027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:39.7011291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:39.7211182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:34:39.7212981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:34:39.7216154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:39.7312806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:41.4699339Z ok (3.610s) 2022-03-17T21:34:41.4700000Z 2022-03-17T21:34:41.4701416Z ---------------------------------------------------------------------- 2022-03-17T21:34:41.4702431Z Ran 1 test in 3.610s 2022-03-17T21:34:41.4702896Z 2022-03-17T21:34:41.4703136Z OK 2022-03-17T21:34:41.4703517Z 2022-03-17T21:34:41.4703888Z Generating XML reports... 2022-03-17T21:34:41.4782454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213437.xml 2022-03-17T21:34:44.1110011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:44.1122074Z 2022-03-17T21:34:44.1122199Z Running tests... 2022-03-17T21:34:44.1122746Z ---------------------------------------------------------------------- 2022-03-17T21:34:44.7099212Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64556 2022-03-17T21:34:44.7131688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64557 2022-03-17T21:34:44.7133359Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:45.8876083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:45.8876548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:45.8877519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:45.8980231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:45.8984995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:45.9993068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:46.0202184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:34:46.0202908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:46.0203463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:34:46.0204233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:47.9233258Z ok (3.811s) 2022-03-17T21:34:47.9233659Z 2022-03-17T21:34:47.9234510Z ---------------------------------------------------------------------- 2022-03-17T21:34:47.9235716Z Ran 1 test in 3.811s 2022-03-17T21:34:47.9236094Z 2022-03-17T21:34:47.9236278Z OK 2022-03-17T21:34:47.9236575Z 2022-03-17T21:34:47.9237086Z Generating XML reports... 2022-03-17T21:34:47.9307193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213444.xml 2022-03-17T21:34:50.4740768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:50.4750738Z 2022-03-17T21:34:50.4751014Z Running tests... 2022-03-17T21:34:50.4751536Z ---------------------------------------------------------------------- 2022-03-17T21:34:51.1028748Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64778 2022-03-17T21:34:51.1060530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64779 2022-03-17T21:34:51.1061903Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:52.2419180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:52.2420457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:52.2422364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:52.2424180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:52.2528183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:52.3431145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:52.3638980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:34:52.3640298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:34:52.3642316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:52.3741123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:54.3162539Z ok (3.840s) 2022-03-17T21:34:54.3163755Z 2022-03-17T21:34:54.3165061Z ---------------------------------------------------------------------- 2022-03-17T21:34:54.3165903Z Ran 1 test in 3.841s 2022-03-17T21:34:54.3166275Z 2022-03-17T21:34:54.3166470Z OK 2022-03-17T21:34:54.3166748Z 2022-03-17T21:34:54.3167044Z Generating XML reports... 2022-03-17T21:34:54.3242847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213450.xml 2022-03-17T21:34:56.9656190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:56.9667756Z 2022-03-17T21:34:56.9668169Z Running tests... 2022-03-17T21:34:56.9669573Z ---------------------------------------------------------------------- 2022-03-17T21:34:57.6198713Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65000 2022-03-17T21:34:57.6247040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65001 2022-03-17T21:34:57.6248600Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:34:58.7768801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:58.7770763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:58.7773015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:58.7871573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:58.7889434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:58.8882288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:00.6319939Z skip (3.665s) 2022-03-17T21:35:00.6320493Z 2022-03-17T21:35:00.6321512Z ---------------------------------------------------------------------- 2022-03-17T21:35:00.6322840Z Ran 1 test in 3.665s 2022-03-17T21:35:00.6323300Z 2022-03-17T21:35:00.6323607Z OK (skipped=1) 2022-03-17T21:35:00.6323986Z 2022-03-17T21:35:00.6324261Z Generating XML reports... 2022-03-17T21:35:00.6392120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213456.xml 2022-03-17T21:35:03.2674232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:03.2687287Z 2022-03-17T21:35:03.2687578Z Running tests... 2022-03-17T21:35:03.2688222Z ---------------------------------------------------------------------- 2022-03-17T21:35:03.8682140Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65210 2022-03-17T21:35:03.8722350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65211 2022-03-17T21:35:03.8723900Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:05.0122445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:05.0123923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:05.0125976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:05.0127800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:05.0131242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:05.1136016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:06.7802383Z skip (3.511s) 2022-03-17T21:35:06.7802931Z 2022-03-17T21:35:06.7803998Z ---------------------------------------------------------------------- 2022-03-17T21:35:06.7804936Z Ran 1 test in 3.511s 2022-03-17T21:35:06.7805408Z 2022-03-17T21:35:06.7805642Z OK (skipped=1) 2022-03-17T21:35:06.7806019Z 2022-03-17T21:35:06.7806271Z Generating XML reports... 2022-03-17T21:35:06.7884152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213503.xml 2022-03-17T21:35:09.4250258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:09.4264403Z 2022-03-17T21:35:09.4265005Z Running tests... 2022-03-17T21:35:09.4266446Z ---------------------------------------------------------------------- 2022-03-17T21:35:10.0559120Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65420 2022-03-17T21:35:10.0596581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65421 2022-03-17T21:35:10.0598174Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:11.1996593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:11.1997482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:11.1998472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:11.1999259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:11.2014164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:11.3009705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:12.9679766Z skip (3.541s) 2022-03-17T21:35:12.9680186Z 2022-03-17T21:35:12.9681087Z ---------------------------------------------------------------------- 2022-03-17T21:35:12.9682060Z Ran 1 test in 3.542s 2022-03-17T21:35:12.9682424Z 2022-03-17T21:35:12.9682673Z OK (skipped=1) 2022-03-17T21:35:12.9683019Z 2022-03-17T21:35:12.9683288Z Generating XML reports... 2022-03-17T21:35:12.9763629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213509.xml 2022-03-17T21:35:15.6691713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:15.6703477Z 2022-03-17T21:35:15.6703765Z Running tests... 2022-03-17T21:35:15.6704909Z ---------------------------------------------------------------------- 2022-03-17T21:35:16.2659398Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65630 2022-03-17T21:35:16.2705919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65631 2022-03-17T21:35:16.2707280Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:17.4189955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:17.4191287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:17.4193349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:17.4195199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:17.4298390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:17.5211663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:19.5792140Z skip (3.908s) 2022-03-17T21:35:19.5792792Z 2022-03-17T21:35:19.5794127Z ---------------------------------------------------------------------- 2022-03-17T21:35:19.5795057Z Ran 1 test in 3.909s 2022-03-17T21:35:19.5795531Z 2022-03-17T21:35:19.5795817Z OK (skipped=1) 2022-03-17T21:35:19.5796186Z 2022-03-17T21:35:19.5796561Z Generating XML reports... 2022-03-17T21:35:19.5877364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213515.xml 2022-03-17T21:35:22.2306382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:22.2321159Z 2022-03-17T21:35:22.2321486Z Running tests... 2022-03-17T21:35:22.2322115Z ---------------------------------------------------------------------- 2022-03-17T21:35:22.8608839Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65840 2022-03-17T21:35:22.8683524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65841 2022-03-17T21:35:22.8685192Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:24.0274954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:24.0276402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:24.0278877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:24.0280640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:24.0385815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:24.1285410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:25.9786592Z ok (3.746s) 2022-03-17T21:35:25.9787172Z 2022-03-17T21:35:25.9788166Z ---------------------------------------------------------------------- 2022-03-17T21:35:25.9789096Z Ran 1 test in 3.747s 2022-03-17T21:35:25.9789515Z 2022-03-17T21:35:25.9789749Z OK 2022-03-17T21:35:25.9790128Z 2022-03-17T21:35:25.9790474Z Generating XML reports... 2022-03-17T21:35:25.9865604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213522.xml 2022-03-17T21:35:28.6095075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:28.6107111Z 2022-03-17T21:35:28.6107372Z Running tests... 2022-03-17T21:35:28.6107817Z ---------------------------------------------------------------------- 2022-03-17T21:35:29.2066498Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66056 2022-03-17T21:35:29.2099833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66057 2022-03-17T21:35:29.2101232Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:30.3307182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:30.3308850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:30.3310200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:30.3311940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:30.3415214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:30.4323089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:32.2200098Z ok (3.609s) 2022-03-17T21:35:32.2200517Z 2022-03-17T21:35:32.2201818Z ---------------------------------------------------------------------- 2022-03-17T21:35:32.2202809Z Ran 1 test in 3.609s 2022-03-17T21:35:32.2203194Z 2022-03-17T21:35:32.2203374Z OK 2022-03-17T21:35:32.2203645Z 2022-03-17T21:35:32.2203917Z Generating XML reports... 2022-03-17T21:35:32.2284644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213528.xml 2022-03-17T21:35:34.8304622Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:34.8315562Z 2022-03-17T21:35:34.8316066Z Running tests... 2022-03-17T21:35:34.8317265Z ---------------------------------------------------------------------- 2022-03-17T21:35:34.8335952Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:35:34.8336975Z 2022-03-17T21:35:34.8337892Z ---------------------------------------------------------------------- 2022-03-17T21:35:34.8338701Z Ran 1 test in 0.002s 2022-03-17T21:35:34.8339067Z 2022-03-17T21:35:34.8339306Z OK (skipped=1) 2022-03-17T21:35:34.8339664Z 2022-03-17T21:35:34.8339937Z Generating XML reports... 2022-03-17T21:35:34.8374806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213534.xml 2022-03-17T21:35:37.4362206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:37.4377124Z 2022-03-17T21:35:37.4377249Z Running tests... 2022-03-17T21:35:37.4377768Z ---------------------------------------------------------------------- 2022-03-17T21:35:38.0748229Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66339 2022-03-17T21:35:38.0781665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66340 2022-03-17T21:35:38.0782999Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:39.2509493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:39.2510737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:39.2512668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:39.2611409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:39.2625962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:39.3617926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:41.3885452Z ok (3.950s) 2022-03-17T21:35:41.3886155Z 2022-03-17T21:35:41.3887783Z ---------------------------------------------------------------------- 2022-03-17T21:35:41.3888683Z Ran 1 test in 3.951s 2022-03-17T21:35:41.3889055Z 2022-03-17T21:35:41.3889240Z OK 2022-03-17T21:35:41.3889533Z 2022-03-17T21:35:41.3889816Z Generating XML reports... 2022-03-17T21:35:41.3968107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213537.xml 2022-03-17T21:35:44.0709643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:44.0725558Z 2022-03-17T21:35:44.0725880Z Running tests... 2022-03-17T21:35:44.0726584Z ---------------------------------------------------------------------- 2022-03-17T21:35:44.6988120Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66555 2022-03-17T21:35:44.7028444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66556 2022-03-17T21:35:44.7029771Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:45.8385985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:45.8386570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:45.8387428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:45.8388248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:45.8499298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:45.9402327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:47.8119087Z ok (3.739s) 2022-03-17T21:35:47.8119449Z 2022-03-17T21:35:47.8120344Z ---------------------------------------------------------------------- 2022-03-17T21:35:47.8121403Z Ran 1 test in 3.739s 2022-03-17T21:35:47.8121772Z 2022-03-17T21:35:47.8122132Z OK 2022-03-17T21:35:47.8122420Z 2022-03-17T21:35:47.8122692Z Generating XML reports... 2022-03-17T21:35:47.8203553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213544.xml 2022-03-17T21:35:50.4184413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:50.4246178Z 2022-03-17T21:35:50.4252863Z Running tests... 2022-03-17T21:35:50.4259274Z ---------------------------------------------------------------------- 2022-03-17T21:35:50.4267832Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.006s) 2022-03-17T21:35:50.4268112Z 2022-03-17T21:35:50.4268497Z ---------------------------------------------------------------------- 2022-03-17T21:35:50.4268891Z Ran 1 test in 0.006s 2022-03-17T21:35:50.4269068Z 2022-03-17T21:35:50.4269163Z OK (skipped=1) 2022-03-17T21:35:50.4269335Z 2022-03-17T21:35:50.4269468Z Generating XML reports... 2022-03-17T21:35:50.4320497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213550.xml 2022-03-17T21:35:53.2030728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:53.2040681Z 2022-03-17T21:35:53.2040936Z Running tests... 2022-03-17T21:35:53.2041373Z ---------------------------------------------------------------------- 2022-03-17T21:35:53.2062083Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:35:53.2062663Z 2022-03-17T21:35:53.2063820Z ---------------------------------------------------------------------- 2022-03-17T21:35:53.2064670Z Ran 1 test in 0.002s 2022-03-17T21:35:53.2065052Z 2022-03-17T21:35:53.2065308Z OK (skipped=1) 2022-03-17T21:35:53.2065650Z 2022-03-17T21:35:53.2065925Z Generating XML reports... 2022-03-17T21:35:53.2101767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213553.xml 2022-03-17T21:35:55.8116859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:55.8123996Z 2022-03-17T21:35:55.8124566Z Running tests... 2022-03-17T21:35:55.8125694Z ---------------------------------------------------------------------- 2022-03-17T21:35:56.4349465Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66905 2022-03-17T21:35:56.4380200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66906 2022-03-17T21:35:56.4381510Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:35:57.5849516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:57.5849995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:57.5850808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:57.5950292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:57.5957350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:57.6961995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:59.5481249Z ok (3.735s) 2022-03-17T21:35:59.5481662Z 2022-03-17T21:35:59.5482696Z ---------------------------------------------------------------------- 2022-03-17T21:35:59.5483500Z Ran 1 test in 3.736s 2022-03-17T21:35:59.5483863Z 2022-03-17T21:35:59.5484073Z OK 2022-03-17T21:35:59.5487738Z 2022-03-17T21:35:59.5488398Z Generating XML reports... 2022-03-17T21:35:59.5565229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213555.xml 2022-03-17T21:36:02.1634348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:02.1647967Z 2022-03-17T21:36:02.1648218Z Running tests... 2022-03-17T21:36:02.1648741Z ---------------------------------------------------------------------- 2022-03-17T21:36:02.7614057Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67121 2022-03-17T21:36:02.7662378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67122 2022-03-17T21:36:02.7663716Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:03.9098885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:03.9099955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:03.9101545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:03.9203145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:03.9206651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:04.0212177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:05.8763516Z ok (3.711s) 2022-03-17T21:36:05.8763987Z 2022-03-17T21:36:05.8765046Z ---------------------------------------------------------------------- 2022-03-17T21:36:05.8765802Z Ran 1 test in 3.711s 2022-03-17T21:36:05.8766217Z 2022-03-17T21:36:05.8766408Z OK 2022-03-17T21:36:05.8766713Z 2022-03-17T21:36:05.8767006Z Generating XML reports... 2022-03-17T21:36:05.8845250Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213602.xml 2022-03-17T21:36:08.4352912Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:08.4363574Z 2022-03-17T21:36:08.4363863Z Running tests... 2022-03-17T21:36:08.4364994Z ---------------------------------------------------------------------- 2022-03-17T21:36:09.0469354Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67337 2022-03-17T21:36:09.0502271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67338 2022-03-17T21:36:09.0503602Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:10.2028990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:10.2030969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:10.2032332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:10.2034043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:10.2140812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:10.3047147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:12.1577706Z ok (3.721s) 2022-03-17T21:36:12.1578169Z 2022-03-17T21:36:12.1579281Z ---------------------------------------------------------------------- 2022-03-17T21:36:12.1580241Z Ran 1 test in 3.721s 2022-03-17T21:36:12.1580730Z 2022-03-17T21:36:12.1580901Z OK 2022-03-17T21:36:12.1581199Z 2022-03-17T21:36:12.1581488Z Generating XML reports... 2022-03-17T21:36:12.1660689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213608.xml 2022-03-17T21:36:14.8466962Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:14.8480014Z 2022-03-17T21:36:14.8480449Z Running tests... 2022-03-17T21:36:14.8482801Z ---------------------------------------------------------------------- 2022-03-17T21:36:15.4836140Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67547 2022-03-17T21:36:15.4872044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67548 2022-03-17T21:36:15.4873652Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:16.6549824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:16.6550404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:16.6551272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:16.6552066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:16.6553648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:16.6554188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:18.3967914Z ok (3.548s) 2022-03-17T21:36:18.3968299Z 2022-03-17T21:36:18.3969193Z ---------------------------------------------------------------------- 2022-03-17T21:36:18.3969970Z Ran 1 test in 3.549s 2022-03-17T21:36:18.3970372Z 2022-03-17T21:36:18.3970579Z OK 2022-03-17T21:36:18.3970882Z 2022-03-17T21:36:18.3971172Z Generating XML reports... 2022-03-17T21:36:18.4052749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213614.xml 2022-03-17T21:36:21.0968321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:21.0979709Z 2022-03-17T21:36:21.0980061Z Running tests... 2022-03-17T21:36:21.0980608Z ---------------------------------------------------------------------- 2022-03-17T21:36:21.0996472Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:36:21.0996811Z 2022-03-17T21:36:21.0997276Z ---------------------------------------------------------------------- 2022-03-17T21:36:21.0997697Z Ran 1 test in 0.002s 2022-03-17T21:36:21.0997906Z 2022-03-17T21:36:21.0998040Z OK (skipped=1) 2022-03-17T21:36:21.0998246Z 2022-03-17T21:36:21.0998614Z Generating XML reports... 2022-03-17T21:36:21.1034644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213621.xml 2022-03-17T21:36:23.7102076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:23.7112925Z 2022-03-17T21:36:23.7113300Z Running tests... 2022-03-17T21:36:23.7114694Z ---------------------------------------------------------------------- 2022-03-17T21:36:23.7130428Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:36:23.7131308Z 2022-03-17T21:36:23.7132301Z ---------------------------------------------------------------------- 2022-03-17T21:36:23.7133270Z Ran 1 test in 0.002s 2022-03-17T21:36:23.7133748Z 2022-03-17T21:36:23.7133970Z OK (skipped=1) 2022-03-17T21:36:23.7134329Z 2022-03-17T21:36:23.7134602Z Generating XML reports... 2022-03-17T21:36:23.7169342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213623.xml 2022-03-17T21:36:26.3161545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:26.3173690Z 2022-03-17T21:36:26.3174269Z Running tests... 2022-03-17T21:36:26.3175723Z ---------------------------------------------------------------------- 2022-03-17T21:36:26.9135873Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67897 2022-03-17T21:36:26.9182263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67898 2022-03-17T21:36:26.9183617Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:28.0710137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:28.0712032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:28.0713322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:28.0714977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:28.0723174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:28.1725338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:28.1846349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:36:28.1847807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:36:28.1849870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:36:28.1851627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:36:30.1282383Z ok (3.810s) 2022-03-17T21:36:30.1282868Z 2022-03-17T21:36:30.1283871Z ---------------------------------------------------------------------- 2022-03-17T21:36:30.1284822Z Ran 1 test in 3.811s 2022-03-17T21:36:30.1285269Z 2022-03-17T21:36:30.1285510Z OK 2022-03-17T21:36:30.1285984Z 2022-03-17T21:36:30.1286253Z Generating XML reports... 2022-03-17T21:36:30.1365768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213626.xml 2022-03-17T21:36:32.7299773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:32.7310995Z 2022-03-17T21:36:32.7311371Z Running tests... 2022-03-17T21:36:32.7312770Z ---------------------------------------------------------------------- 2022-03-17T21:36:33.3570573Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68119 2022-03-17T21:36:33.3604851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68120 2022-03-17T21:36:33.3606396Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:34.4933342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:34.4934829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:34.4936901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:34.4938640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:34.5046870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:34.5952135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:36.3695314Z skip (3.638s) 2022-03-17T21:36:36.3695807Z 2022-03-17T21:36:36.3696895Z ---------------------------------------------------------------------- 2022-03-17T21:36:36.3697869Z Ran 1 test in 3.638s 2022-03-17T21:36:36.3698324Z 2022-03-17T21:36:36.3698610Z OK (skipped=1) 2022-03-17T21:36:36.3699029Z 2022-03-17T21:36:36.3699357Z Generating XML reports... 2022-03-17T21:36:36.3781710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213632.xml 2022-03-17T21:36:39.0797482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:39.0809806Z 2022-03-17T21:36:39.0810513Z Running tests... 2022-03-17T21:36:39.0811932Z ---------------------------------------------------------------------- 2022-03-17T21:36:39.6830427Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68329 2022-03-17T21:36:39.6863387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68330 2022-03-17T21:36:39.6864951Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:40.8773640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:40.8775133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:40.8777268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:40.8779026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:40.8890881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:40.9789440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:42.6940135Z ok (3.613s) 2022-03-17T21:36:42.6940636Z 2022-03-17T21:36:42.6941726Z ---------------------------------------------------------------------- 2022-03-17T21:36:42.6942501Z Ran 1 test in 3.613s 2022-03-17T21:36:42.6942848Z 2022-03-17T21:36:42.6943052Z OK 2022-03-17T21:36:42.6943356Z 2022-03-17T21:36:42.6943639Z Generating XML reports... 2022-03-17T21:36:42.7025392Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213639.xml 2022-03-17T21:36:45.2786286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:45.2797313Z 2022-03-17T21:36:45.2797944Z Running tests... 2022-03-17T21:36:45.2799381Z ---------------------------------------------------------------------- 2022-03-17T21:36:45.8758585Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68539 2022-03-17T21:36:45.8790888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68540 2022-03-17T21:36:45.8792495Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:47.0489044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:47.0490549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:47.0492662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:47.0494495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:47.0601419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:47.1511057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:48.9875678Z ok (3.707s) 2022-03-17T21:36:48.9876294Z 2022-03-17T21:36:48.9877312Z ---------------------------------------------------------------------- 2022-03-17T21:36:48.9878277Z Ran 1 test in 3.708s 2022-03-17T21:36:48.9878705Z 2022-03-17T21:36:48.9878942Z OK 2022-03-17T21:36:48.9879312Z 2022-03-17T21:36:48.9880043Z Generating XML reports... 2022-03-17T21:36:48.9956374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213645.xml 2022-03-17T21:36:51.5429399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:51.5440286Z 2022-03-17T21:36:51.5440615Z Running tests... 2022-03-17T21:36:51.5441155Z ---------------------------------------------------------------------- 2022-03-17T21:36:52.1438554Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68749 2022-03-17T21:36:52.1475361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68750 2022-03-17T21:36:52.1476795Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:53.2952381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:53.2952982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:53.2953864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:53.2954663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:53.2965595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:36:53.2966141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:55.0570870Z ok (3.512s) 2022-03-17T21:36:55.0571297Z 2022-03-17T21:36:55.0572537Z ---------------------------------------------------------------------- 2022-03-17T21:36:55.0573485Z Ran 1 test in 3.513s 2022-03-17T21:36:55.0573831Z 2022-03-17T21:36:55.0574013Z OK 2022-03-17T21:36:55.0574301Z 2022-03-17T21:36:55.0574603Z Generating XML reports... 2022-03-17T21:36:55.0653877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213651.xml 2022-03-17T21:36:57.6995022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:36:57.7006299Z 2022-03-17T21:36:57.7006805Z Running tests... 2022-03-17T21:36:57.7007974Z ---------------------------------------------------------------------- 2022-03-17T21:36:58.3009134Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68959 2022-03-17T21:36:58.3046111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68960 2022-03-17T21:36:58.3047424Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:36:59.4551479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:36:59.4552053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:59.4553419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:59.4653698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:36:59.4655700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:36:59.5576021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:01.3142562Z ok (3.613s) 2022-03-17T21:37:01.3142964Z 2022-03-17T21:37:01.3144404Z ---------------------------------------------------------------------- 2022-03-17T21:37:01.3145408Z Ran 1 test in 3.614s 2022-03-17T21:37:01.3145866Z 2022-03-17T21:37:01.3146538Z OK 2022-03-17T21:37:01.3146876Z 2022-03-17T21:37:01.3147155Z Generating XML reports... 2022-03-17T21:37:01.3220961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213657.xml 2022-03-17T21:37:03.9048290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:03.9061292Z 2022-03-17T21:37:03.9061517Z Running tests... 2022-03-17T21:37:03.9062054Z ---------------------------------------------------------------------- 2022-03-17T21:37:04.5043405Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69175 2022-03-17T21:37:04.5090843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69176 2022-03-17T21:37:04.5092341Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:05.6645564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:05.6648609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:05.6650860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:05.6749562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:05.6772668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:05.7759746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:07.7182770Z ok (3.812s) 2022-03-17T21:37:07.7183195Z 2022-03-17T21:37:07.7184351Z ---------------------------------------------------------------------- 2022-03-17T21:37:07.7185211Z Ran 1 test in 3.812s 2022-03-17T21:37:07.7185605Z 2022-03-17T21:37:07.7185802Z OK 2022-03-17T21:37:07.7189627Z 2022-03-17T21:37:07.7190204Z Generating XML reports... 2022-03-17T21:37:07.7260683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213703.xml 2022-03-17T21:37:10.3233118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:10.3243208Z 2022-03-17T21:37:10.3243682Z Running tests... 2022-03-17T21:37:10.3245343Z ---------------------------------------------------------------------- 2022-03-17T21:37:10.9217883Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69391 2022-03-17T21:37:10.9249346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69392 2022-03-17T21:37:10.9250654Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:12.0273526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:12.0274544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:12.0275148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:12.0275925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:12.0380031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:12.1292086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:13.8330949Z ok (3.508s) 2022-03-17T21:37:13.8331997Z 2022-03-17T21:37:13.8333428Z ---------------------------------------------------------------------- 2022-03-17T21:37:13.8334215Z Ran 1 test in 3.509s 2022-03-17T21:37:13.8335059Z 2022-03-17T21:37:13.8335255Z OK 2022-03-17T21:37:13.8335553Z 2022-03-17T21:37:13.8335829Z Generating XML reports... 2022-03-17T21:37:13.8415394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213710.xml 2022-03-17T21:37:16.3779473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:16.3790948Z 2022-03-17T21:37:16.3791813Z Running tests... 2022-03-17T21:37:16.3792388Z ---------------------------------------------------------------------- 2022-03-17T21:37:16.3807619Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:37:16.3808376Z 2022-03-17T21:37:16.3809339Z ---------------------------------------------------------------------- 2022-03-17T21:37:16.3810132Z Ran 1 test in 0.002s 2022-03-17T21:37:16.3810495Z 2022-03-17T21:37:16.3810728Z OK (skipped=1) 2022-03-17T21:37:16.3811082Z 2022-03-17T21:37:16.3811355Z Generating XML reports... 2022-03-17T21:37:16.3847179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213716.xml 2022-03-17T21:37:18.9959418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:18.9969979Z 2022-03-17T21:37:18.9970316Z Running tests... 2022-03-17T21:37:18.9971490Z ---------------------------------------------------------------------- 2022-03-17T21:37:18.9986375Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:37:18.9987137Z 2022-03-17T21:37:18.9988185Z ---------------------------------------------------------------------- 2022-03-17T21:37:18.9988974Z Ran 1 test in 0.002s 2022-03-17T21:37:18.9989330Z 2022-03-17T21:37:18.9989566Z OK (skipped=1) 2022-03-17T21:37:18.9989917Z 2022-03-17T21:37:18.9990185Z Generating XML reports... 2022-03-17T21:37:19.0028513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213718.xml 2022-03-17T21:37:21.4759336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:21.4771026Z 2022-03-17T21:37:21.4771250Z Running tests... 2022-03-17T21:37:21.4771773Z ---------------------------------------------------------------------- 2022-03-17T21:37:21.4791746Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:37:21.4792030Z 2022-03-17T21:37:21.4792692Z ---------------------------------------------------------------------- 2022-03-17T21:37:21.4793019Z Ran 1 test in 0.002s 2022-03-17T21:37:21.4793190Z 2022-03-17T21:37:21.4793296Z OK (skipped=1) 2022-03-17T21:37:21.4793463Z 2022-03-17T21:37:21.4793576Z Generating XML reports... 2022-03-17T21:37:21.4831971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213721.xml 2022-03-17T21:37:24.0312833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:24.0324021Z 2022-03-17T21:37:24.0324364Z Running tests... 2022-03-17T21:37:24.0324853Z ---------------------------------------------------------------------- 2022-03-17T21:37:24.6719304Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69808 2022-03-17T21:37:24.6755613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69809 2022-03-17T21:37:24.6756957Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:25.8119657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:25.8120396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:25.8121282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:25.8122474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:25.8124772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:25.8125622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:27.5842236Z ok (3.551s) 2022-03-17T21:37:27.5842681Z 2022-03-17T21:37:27.5843814Z ---------------------------------------------------------------------- 2022-03-17T21:37:27.5844598Z Ran 1 test in 3.552s 2022-03-17T21:37:27.5844965Z 2022-03-17T21:37:27.5845132Z OK 2022-03-17T21:37:27.5845432Z 2022-03-17T21:37:27.5845705Z Generating XML reports... 2022-03-17T21:37:27.5926051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213724.xml 2022-03-17T21:37:30.1641781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:30.1654527Z 2022-03-17T21:37:30.1654776Z Running tests... 2022-03-17T21:37:30.1655306Z ---------------------------------------------------------------------- 2022-03-17T21:37:30.7625952Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70024 2022-03-17T21:37:30.7659386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70025 2022-03-17T21:37:30.7660683Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:31.9337439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:31.9338680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:31.9340640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:31.9441603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:31.9442865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:32.0360352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:33.7737785Z ok (3.608s) 2022-03-17T21:37:33.7738201Z 2022-03-17T21:37:33.7739021Z ---------------------------------------------------------------------- 2022-03-17T21:37:33.7739815Z Ran 1 test in 3.608s 2022-03-17T21:37:33.7740182Z 2022-03-17T21:37:33.7740379Z OK 2022-03-17T21:37:33.7740670Z 2022-03-17T21:37:33.7740949Z Generating XML reports... 2022-03-17T21:37:33.7820639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213730.xml 2022-03-17T21:37:36.3464964Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:36.3478155Z 2022-03-17T21:37:36.3478720Z Running tests... 2022-03-17T21:37:36.3479323Z ---------------------------------------------------------------------- 2022-03-17T21:37:36.9445427Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70234 2022-03-17T21:37:36.9488081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70235 2022-03-17T21:37:36.9490351Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:38.0953806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:38.0954255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:38.0955311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:38.0956105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:38.1060737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:38.1967412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:39.9589909Z ok (3.611s) 2022-03-17T21:37:39.9590267Z 2022-03-17T21:37:39.9591116Z ---------------------------------------------------------------------- 2022-03-17T21:37:39.9591922Z Ran 1 test in 3.611s 2022-03-17T21:37:39.9592300Z 2022-03-17T21:37:39.9592469Z OK 2022-03-17T21:37:39.9592760Z 2022-03-17T21:37:39.9593046Z Generating XML reports... 2022-03-17T21:37:39.9665619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213736.xml 2022-03-17T21:37:42.5144377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:42.5156489Z 2022-03-17T21:37:42.5156935Z Running tests... 2022-03-17T21:37:42.5157362Z ---------------------------------------------------------------------- 2022-03-17T21:37:43.1465435Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70450 2022-03-17T21:37:43.1498457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70451 2022-03-17T21:37:43.1499772Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:44.3056687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:44.3057926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:44.3059777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:44.3061517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:44.3063223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:44.3064916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:46.0594556Z ok (3.543s) 2022-03-17T21:37:46.0594942Z 2022-03-17T21:37:46.0595785Z ---------------------------------------------------------------------- 2022-03-17T21:37:46.0596547Z Ran 1 test in 3.544s 2022-03-17T21:37:46.0596918Z 2022-03-17T21:37:46.0598507Z OK 2022-03-17T21:37:46.0598892Z 2022-03-17T21:37:46.0599272Z Generating XML reports... 2022-03-17T21:37:46.0664217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213742.xml 2022-03-17T21:37:48.6087511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:48.6097024Z 2022-03-17T21:37:48.6097270Z Running tests... 2022-03-17T21:37:48.6097705Z ---------------------------------------------------------------------- 2022-03-17T21:37:49.2079992Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70666 2022-03-17T21:37:49.2119583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70667 2022-03-17T21:37:49.2120682Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:50.3544419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:50.3544882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:50.3545901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:50.3546696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:50.3550538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:50.3551176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:52.1212533Z ok (3.511s) 2022-03-17T21:37:52.1212911Z 2022-03-17T21:37:52.1213779Z ---------------------------------------------------------------------- 2022-03-17T21:37:52.1214530Z Ran 1 test in 3.511s 2022-03-17T21:37:52.1214892Z 2022-03-17T21:37:52.1215086Z OK 2022-03-17T21:37:52.1215374Z 2022-03-17T21:37:52.1215631Z Generating XML reports... 2022-03-17T21:37:52.1296508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213748.xml 2022-03-17T21:37:54.8144457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:37:54.8159110Z 2022-03-17T21:37:54.8159408Z Running tests... 2022-03-17T21:37:54.8159898Z ---------------------------------------------------------------------- 2022-03-17T21:37:55.4525710Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71062 2022-03-17T21:37:55.4569314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71063 2022-03-17T21:37:55.4570657Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:37:56.5353647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:37:56.5354112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:37:56.5354821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:56.5456709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:37:56.5459119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:37:56.6371589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:37:58.8674019Z ok (4.051s) 2022-03-17T21:37:58.8674605Z 2022-03-17T21:37:58.8675901Z ---------------------------------------------------------------------- 2022-03-17T21:37:58.8676858Z Ran 1 test in 4.051s 2022-03-17T21:37:58.8677297Z 2022-03-17T21:37:58.8677506Z OK 2022-03-17T21:37:58.8677860Z 2022-03-17T21:37:58.8678246Z Generating XML reports... 2022-03-17T21:37:58.8747383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213754.xml 2022-03-17T21:38:01.4295110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:38:01.4306662Z 2022-03-17T21:38:01.4306955Z Running tests... 2022-03-17T21:38:01.4307612Z ---------------------------------------------------------------------- 2022-03-17T21:38:02.0504155Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71458 2022-03-17T21:38:02.0537999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71459 2022-03-17T21:38:02.0538713Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:38:03.2503565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:38:03.2505801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:03.2507623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:38:03.2509330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:03.2611966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:38:03.3529245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:38:15.5811585Z ok (14.150s) 2022-03-17T21:38:15.5812201Z 2022-03-17T21:38:15.5813199Z ---------------------------------------------------------------------- 2022-03-17T21:38:15.5814052Z Ran 1 test in 14.150s 2022-03-17T21:38:15.5814571Z 2022-03-17T21:38:15.5814788Z OK 2022-03-17T21:38:15.5815086Z 2022-03-17T21:38:15.5815455Z Generating XML reports... 2022-03-17T21:38:15.5891090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213801.xml 2022-03-17T21:38:18.2302995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:38:18.2315123Z 2022-03-17T21:38:18.2315621Z Running tests... 2022-03-17T21:38:18.2316790Z ---------------------------------------------------------------------- 2022-03-17T21:38:18.8543958Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71672 2022-03-17T21:38:18.8583512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71673 2022-03-17T21:38:18.8584842Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:38:20.0051664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:38:20.0053974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:20.0055420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:38:20.0057104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:20.0166077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:38:20.1070806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:38:21.8665830Z ok (3.635s) 2022-03-17T21:38:21.8666363Z 2022-03-17T21:38:21.8667420Z ---------------------------------------------------------------------- 2022-03-17T21:38:21.8668386Z Ran 1 test in 3.635s 2022-03-17T21:38:21.8668828Z 2022-03-17T21:38:21.8669066Z OK 2022-03-17T21:38:21.8669408Z 2022-03-17T21:38:21.8669810Z Generating XML reports... 2022-03-17T21:38:21.8743069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213818.xml 2022-03-17T21:38:24.5597675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:38:24.5608610Z 2022-03-17T21:38:24.5609201Z Running tests... 2022-03-17T21:38:24.5610613Z ---------------------------------------------------------------------- 2022-03-17T21:38:25.1852717Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71948 2022-03-17T21:38:25.1904039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71949 2022-03-17T21:38:25.1905633Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:38:26.3408657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:38:26.3410767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:38:26.3412872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:26.3508806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:26.3511837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:38:26.4435031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:38:28.5009566Z ok (3.939s) 2022-03-17T21:38:28.5010184Z 2022-03-17T21:38:28.5011417Z ---------------------------------------------------------------------- 2022-03-17T21:38:28.5012427Z Ran 1 test in 3.940s 2022-03-17T21:38:28.5012806Z 2022-03-17T21:38:28.5013003Z OK 2022-03-17T21:38:28.5013282Z 2022-03-17T21:38:28.5013575Z Generating XML reports... 2022-03-17T21:38:28.5089175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213824.xml 2022-03-17T21:38:31.2079127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:38:31.2089467Z 2022-03-17T21:38:31.2090068Z Running tests... 2022-03-17T21:38:31.2091485Z ---------------------------------------------------------------------- 2022-03-17T21:38:31.8048008Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72158 2022-03-17T21:38:31.8089932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72159 2022-03-17T21:38:31.8091480Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:38:32.9492744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:38:32.9495043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:32.9496372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:38:32.9498034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:32.9602767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:38:33.0513687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:38:42.5018471Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:38:42.5527279Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:38:45.2415612Z ok (14.032s) 2022-03-17T21:38:45.2416108Z 2022-03-17T21:38:45.2417293Z ---------------------------------------------------------------------- 2022-03-17T21:38:45.2418231Z Ran 1 test in 14.032s 2022-03-17T21:38:45.2419323Z 2022-03-17T21:38:45.2419517Z OK 2022-03-17T21:38:45.2419812Z 2022-03-17T21:38:45.2420076Z Generating XML reports... 2022-03-17T21:38:45.2497501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213831.xml 2022-03-17T21:38:47.8839806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:38:47.8851597Z 2022-03-17T21:38:47.8852227Z Running tests... 2022-03-17T21:38:47.8853676Z ---------------------------------------------------------------------- 2022-03-17T21:38:48.5173560Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72372 2022-03-17T21:38:48.5214828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72373 2022-03-17T21:38:48.5216369Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:38:49.6603819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:38:49.6605280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:38:49.6607329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:49.6609136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:49.6616600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:38:49.7617054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:38:51.4309423Z skip (3.545s) 2022-03-17T21:38:51.4309923Z 2022-03-17T21:38:51.4310951Z ---------------------------------------------------------------------- 2022-03-17T21:38:51.4311906Z Ran 1 test in 3.546s 2022-03-17T21:38:51.4312371Z 2022-03-17T21:38:51.4312626Z OK (skipped=1) 2022-03-17T21:38:51.4312981Z 2022-03-17T21:38:51.4313236Z Generating XML reports... 2022-03-17T21:38:51.4384305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213847.xml 2022-03-17T21:38:54.0373354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:38:54.0385313Z 2022-03-17T21:38:54.0386221Z Running tests... 2022-03-17T21:38:54.0387676Z ---------------------------------------------------------------------- 2022-03-17T21:38:54.6357332Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72582 2022-03-17T21:38:54.6394396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72583 2022-03-17T21:38:54.6395958Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:38:55.7399656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:38:55.7413571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:38:55.7414547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:55.7505074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:38:55.7513491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:38:55.8435023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:38:57.6521315Z skip (3.613s) 2022-03-17T21:38:57.6521709Z 2022-03-17T21:38:57.6523486Z ---------------------------------------------------------------------- 2022-03-17T21:38:57.6524919Z Ran 1 test in 3.613s 2022-03-17T21:38:57.6525287Z 2022-03-17T21:38:57.6525517Z OK (skipped=1) 2022-03-17T21:38:57.6525872Z 2022-03-17T21:38:57.6526161Z Generating XML reports... 2022-03-17T21:38:57.6603701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213854.xml 2022-03-17T21:39:00.6163155Z Running distributed tests for the gloo backend with file init_method 2022-03-17T21:39:00.6165475Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:39:00.615818] 2022-03-17T21:39:01.7160988Z 2022-03-17T21:39:01.7249263Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T21:39:01.7312798Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7313278Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7327445Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7327930Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7328411Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7328929Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7329466Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7330007Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7330589Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7331201Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7331809Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7332394Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7333129Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7333694Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7334225Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7334758Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7335279Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7335754Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7336212Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7336709Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7337222Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7337759Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7338215Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7338616Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7339006Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7339383Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7339773Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7340161Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7340525Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7340880Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7341225Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7341591Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7341956Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7342298Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7342666Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7343045Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7343428Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7343878Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7344268Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7344681Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7345084Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7345478Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7345868Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7346255Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7346649Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7347017Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7347419Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7347820Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7348199Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7348576Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7348951Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7349343Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7349750Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7350136Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7350516Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7350870Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7351232Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7351592Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7351955Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7352305Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7352633Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7352981Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7353341Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7353721Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7354084Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7354421Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7354772Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7355120Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7355474Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7355841Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7356201Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7356554Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7356883Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7357232Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7357574Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7357941Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7358306Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7358651Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7359004Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7359357Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7359819Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7360218Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7360612Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7361019Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7361418Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7361889Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7362293Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7362681Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7363149Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7363635Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7364141Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7364644Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7365140Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7365637Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7366189Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7366660Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7367086Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7367500Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7367905Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7368296Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7368718Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7369144Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7369563Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7369978Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7370395Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7370856Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7371307Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7371729Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7372168Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7372619Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7373082Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7373537Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7373987Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7374428Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7374876Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7375332Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7375760Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7376158Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7376582Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7377002Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7377435Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7378005Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7378518Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7378990Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7379394Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7379770Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7380137Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7380534Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7380949Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7381362Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7381764Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7382144Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7382547Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7382921Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7383251Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7383608Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7383969Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7384386Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7384766Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7385150Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7385555Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7385933Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7386358Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7386823Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7387336Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7387908Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7388488Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7389052Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7389624Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7390200Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7390747Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7391304Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7391801Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7392264Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7392673Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7393060Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7393426Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7393785Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7394150Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7394540Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7394928Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7395354Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7395778Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7396156Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7396511Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7396876Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7397278Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7397667Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7398052Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7398430Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7398808Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7399247Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7399620Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7399999Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7400381Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7400748Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7401118Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7401497Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7402060Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7402457Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7402796Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7403161Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7403547Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7403925Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7404276Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7404599Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7404938Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7405274Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7405617Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7405945Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7406300Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7406654Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7406977Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7407309Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7407647Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7408014Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7408375Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7408699Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7409016Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7409427Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7409797Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7410176Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7410588Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7411001Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7411384Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7411771Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7412174Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7412554Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7412948Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7413323Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7413711Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7414077Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7414461Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7414818Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7415171Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7415652Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7416088Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7416522Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7416936Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7417350Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7417766Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7418162Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7418539Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7418924Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7419295Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7419657Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7420011Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7420382Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7420733Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7421069Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7421426Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7421771Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7422111Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7422447Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7422776Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7423129Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7423456Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7423792Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7424151Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7424492Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7424824Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7425189Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7425539Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7425867Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7426219Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7426585Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7426919Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7427270Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7427605Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7427947Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7428334Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7428726Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7429123Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7429473Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7429843Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7430237Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7430601Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7430968Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7431378Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7431777Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7432158Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7432519Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7432891Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7433249Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7433595Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7433959Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7434378Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:01.7434789Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:39:02.7667913Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:02.7679609Z 2022-03-17T21:39:02.7680351Z Running tests... 2022-03-17T21:39:02.7681526Z ---------------------------------------------------------------------- 2022-03-17T21:39:03.3689974Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72859 2022-03-17T21:39:03.3737971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72860 2022-03-17T21:39:03.3739304Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:39:04.4851741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:39:04.4852228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:39:04.4853019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:04.4853811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:04.4960298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:39:04.5866562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:39:04.8943773Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:04.8946154Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:04.9833102Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:04.9835920Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:05.3843695Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:05.3845160Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:05.3847630Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:05.3850134Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:05.3950543Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:05.3953264Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:05.3954999Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:05.3957234Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:05.4060536Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:05.4061999Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:39:05.4064390Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:05.4066894Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:39:07.0850545Z ok (4.316s) 2022-03-17T21:39:07.0850942Z 2022-03-17T21:39:07.0851869Z ---------------------------------------------------------------------- 2022-03-17T21:39:07.0852618Z Ran 1 test in 4.317s 2022-03-17T21:39:07.0852979Z 2022-03-17T21:39:07.0853174Z OK 2022-03-17T21:39:07.0853462Z 2022-03-17T21:39:07.0853756Z Generating XML reports... 2022-03-17T21:39:07.0935216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213902.xml 2022-03-17T21:39:09.9093553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:09.9106377Z 2022-03-17T21:39:09.9106752Z Running tests... 2022-03-17T21:39:09.9107195Z ---------------------------------------------------------------------- 2022-03-17T21:39:09.9140348Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T21:39:09.9140747Z 2022-03-17T21:39:09.9141632Z ---------------------------------------------------------------------- 2022-03-17T21:39:09.9142591Z Ran 1 test in 0.004s 2022-03-17T21:39:09.9143045Z 2022-03-17T21:39:09.9143356Z OK (skipped=1) 2022-03-17T21:39:09.9143775Z 2022-03-17T21:39:09.9144119Z Generating XML reports... 2022-03-17T21:39:09.9180782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213909.xml 2022-03-17T21:39:12.6096911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:12.6107198Z 2022-03-17T21:39:12.6107874Z Running tests... 2022-03-17T21:39:12.6109158Z ---------------------------------------------------------------------- 2022-03-17T21:39:13.2125542Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73136 2022-03-17T21:39:13.2199572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73137 2022-03-17T21:39:13.2200881Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:39:14.3474041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:39:14.3476400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:14.3477751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:39:14.3479489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:14.3486618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:39:14.4486667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:39:16.2300856Z ok (3.619s) 2022-03-17T21:39:16.2301370Z 2022-03-17T21:39:16.2302509Z ---------------------------------------------------------------------- 2022-03-17T21:39:16.2303522Z Ran 1 test in 3.619s 2022-03-17T21:39:16.2303979Z 2022-03-17T21:39:16.2304211Z OK 2022-03-17T21:39:16.2304561Z 2022-03-17T21:39:16.2304904Z Generating XML reports... 2022-03-17T21:39:16.2381320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213912.xml 2022-03-17T21:39:18.8122576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:18.8137137Z 2022-03-17T21:39:18.8137557Z Running tests... 2022-03-17T21:39:18.8138207Z ---------------------------------------------------------------------- 2022-03-17T21:39:19.4137123Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73346 2022-03-17T21:39:19.4178891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73347 2022-03-17T21:39:19.4180234Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:39:20.5867146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:39:20.5868629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:39:20.5870756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:20.5872609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:20.5879652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:39:20.5880869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:39:30.4239699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.4241207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.4554863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.4556985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.4675777Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:39:30.4678013Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:39:30.4680943Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:39:30.4683171Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:39:30.6054082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.6055630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.6541661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.6543247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.6805028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.6806534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.8190396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.8191935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.8479995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.8481541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.9960493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:30.9962622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.1382216Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T21:39:31.1384111Z "You've set static_graph to be True, no need to set it again." 2022-03-17T21:39:31.1385825Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T21:39:31.1387305Z "You've set static_graph to be True, no need to set it again." 2022-03-17T21:39:31.1631571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.1633151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.3328897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.3330515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.5027579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.5028952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.6610040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.6611398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.6879641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.6881379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.7283805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.7285194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.8875824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.8877096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.9158737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:31.9160083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.0550395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.0551793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.0828730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.0830053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.2278524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.2279880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.3950231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.3952196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.4562529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.4563888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.6220832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:32.6222222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:35.4512517Z ok (16.637s) 2022-03-17T21:39:35.4513159Z 2022-03-17T21:39:35.4514490Z ---------------------------------------------------------------------- 2022-03-17T21:39:35.4515369Z Ran 1 test in 16.638s 2022-03-17T21:39:35.4515790Z 2022-03-17T21:39:35.4516073Z OK 2022-03-17T21:39:35.4516468Z 2022-03-17T21:39:35.4519119Z Generating XML reports... 2022-03-17T21:39:35.4586775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213918.xml 2022-03-17T21:39:38.3496103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:38.3505364Z 2022-03-17T21:39:38.3506146Z Running tests... 2022-03-17T21:39:38.3507063Z ---------------------------------------------------------------------- 2022-03-17T21:39:38.9823058Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73560 2022-03-17T21:39:38.9875104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73561 2022-03-17T21:39:38.9876698Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:39:40.1591937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:39:40.1593274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:39:40.1595297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:40.1693073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:40.1714613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:39:40.2703258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:39:40.3097228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:40.3097993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:40.3425204Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:39:40.3426150Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:39:40.3429251Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:39:40.3430152Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:39:42.0962780Z ok (3.745s) 2022-03-17T21:39:42.0963169Z 2022-03-17T21:39:42.0964125Z ---------------------------------------------------------------------- 2022-03-17T21:39:42.0965401Z Ran 1 test in 3.745s 2022-03-17T21:39:42.0965771Z 2022-03-17T21:39:42.0965956Z OK 2022-03-17T21:39:42.0966242Z 2022-03-17T21:39:42.0966517Z Generating XML reports... 2022-03-17T21:39:42.1037134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213938.xml 2022-03-17T21:39:44.7487454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:44.7503014Z 2022-03-17T21:39:44.7503278Z Running tests... 2022-03-17T21:39:44.7503715Z ---------------------------------------------------------------------- 2022-03-17T21:39:45.3502789Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73836 2022-03-17T21:39:45.3548154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73837 2022-03-17T21:39:45.3549396Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:39:46.5140866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:39:46.5143099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:46.5144431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:39:46.5146100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:46.5148277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:39:46.6153509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:39:46.6420980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:46.6422173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:39:46.6617807Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:39:46.6619981Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:39:46.6622178Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:39:46.6623741Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:39:48.4634966Z ok (3.713s) 2022-03-17T21:39:48.4635341Z 2022-03-17T21:39:48.4636249Z ---------------------------------------------------------------------- 2022-03-17T21:39:48.4637039Z Ran 1 test in 3.713s 2022-03-17T21:39:48.4637401Z 2022-03-17T21:39:48.4637593Z OK 2022-03-17T21:39:48.4637884Z 2022-03-17T21:39:48.4638140Z Generating XML reports... 2022-03-17T21:39:48.4710739Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213944.xml 2022-03-17T21:39:51.0585622Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:39:51.0594500Z 2022-03-17T21:39:51.0594791Z Running tests... 2022-03-17T21:39:51.0595929Z ---------------------------------------------------------------------- 2022-03-17T21:39:51.6959357Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74112 2022-03-17T21:39:51.7012195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74113 2022-03-17T21:39:51.7013568Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:39:52.8533262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:39:52.8534079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:39:52.8541217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:52.8542399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:39:52.8646485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:39:52.9556145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:40:02.8411854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:02.8413126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:02.8841082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:02.8842381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:02.8997177Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:02.8998746Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:02.9004125Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:02.9005549Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:03.0366192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.0367421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.0773377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.0774259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.2301527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.2302438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.2694324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:03.2695556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:06.2319616Z ok (15.172s) 2022-03-17T21:40:06.2320229Z 2022-03-17T21:40:06.2321337Z ---------------------------------------------------------------------- 2022-03-17T21:40:06.2322326Z Ran 1 test in 15.172s 2022-03-17T21:40:06.2322673Z 2022-03-17T21:40:06.2322861Z OK 2022-03-17T21:40:06.2323153Z 2022-03-17T21:40:06.2323437Z Generating XML reports... 2022-03-17T21:40:06.2405239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213951.xml 2022-03-17T21:40:08.9295433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:40:08.9308192Z 2022-03-17T21:40:08.9308363Z Running tests... 2022-03-17T21:40:08.9308916Z ---------------------------------------------------------------------- 2022-03-17T21:40:09.5490159Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74334 2022-03-17T21:40:09.5540009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74335 2022-03-17T21:40:09.5541400Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:10.6972218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:40:10.6973451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:40:10.6975559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:10.6977567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:10.6978850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:40:10.6980007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:40:11.1051772Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:11.1053244Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:11.3948669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:11.3949904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:11.4200096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:11.4201342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:11.4277490Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:11.4279561Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:11.4303648Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:11.4306046Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:13.0624868Z ok (4.131s) 2022-03-17T21:40:13.0625307Z 2022-03-17T21:40:13.0626239Z ---------------------------------------------------------------------- 2022-03-17T21:40:13.0627002Z Ran 1 test in 4.131s 2022-03-17T21:40:13.0627377Z 2022-03-17T21:40:13.0631095Z OK 2022-03-17T21:40:13.0631532Z 2022-03-17T21:40:13.0631830Z Generating XML reports... 2022-03-17T21:40:13.0664909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214008.xml 2022-03-17T21:40:16.0186595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:40:16.0197635Z 2022-03-17T21:40:16.0198172Z Running tests... 2022-03-17T21:40:16.0199370Z ---------------------------------------------------------------------- 2022-03-17T21:40:16.6159282Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74548 2022-03-17T21:40:16.6190696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74549 2022-03-17T21:40:16.6192061Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:17.7558709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:40:17.7560005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:40:17.7562108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:17.7662196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:17.7663954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:40:17.8579819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:40:18.5696682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:18.5699154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:18.5966774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:18.5968633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:18.6061320Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:18.6063351Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:18.6066144Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:18.6068661Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:20.3301558Z ok (4.310s) 2022-03-17T21:40:20.3302173Z 2022-03-17T21:40:20.3303177Z ---------------------------------------------------------------------- 2022-03-17T21:40:20.3304099Z Ran 1 test in 4.310s 2022-03-17T21:40:20.3304539Z 2022-03-17T21:40:20.3304782Z OK 2022-03-17T21:40:20.3305155Z 2022-03-17T21:40:20.3305418Z Generating XML reports... 2022-03-17T21:40:20.3389068Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214016.xml 2022-03-17T21:40:22.9763828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:40:22.9775361Z 2022-03-17T21:40:22.9776229Z Running tests... 2022-03-17T21:40:22.9777744Z ---------------------------------------------------------------------- 2022-03-17T21:40:23.6140795Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74762 2022-03-17T21:40:23.6187268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74763 2022-03-17T21:40:23.6188977Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:24.7261984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:40:24.7262569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:40:24.7263616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:24.7264415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:24.7371254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:40:24.8276733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:40:25.3578910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:25.3580136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:27.3304215Z ok (4.352s) 2022-03-17T21:40:27.3304612Z 2022-03-17T21:40:27.3305500Z ---------------------------------------------------------------------- 2022-03-17T21:40:27.3306302Z Ran 1 test in 4.353s 2022-03-17T21:40:27.3306682Z 2022-03-17T21:40:27.3306875Z OK 2022-03-17T21:40:27.3307176Z 2022-03-17T21:40:27.3307446Z Generating XML reports... 2022-03-17T21:40:27.3393259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214022.xml 2022-03-17T21:40:29.9878001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:40:29.9887504Z 2022-03-17T21:40:29.9887889Z Running tests... 2022-03-17T21:40:29.9888412Z ---------------------------------------------------------------------- 2022-03-17T21:40:30.5841114Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74976 2022-03-17T21:40:30.5881176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74977 2022-03-17T21:40:30.5882776Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:31.7263654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:40:31.7267192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:40:31.7269187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:31.7366362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:31.7375734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:40:31.8289381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:40:41.7956669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:41.7959630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:41.8362963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:41.8364244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:41.8475733Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:41.8477739Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:41.8514612Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:41.8517298Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:44.7178175Z ok (14.728s) 2022-03-17T21:40:44.7178572Z 2022-03-17T21:40:44.7179422Z ---------------------------------------------------------------------- 2022-03-17T21:40:44.7180204Z Ran 1 test in 14.729s 2022-03-17T21:40:44.7180570Z 2022-03-17T21:40:44.7180770Z OK 2022-03-17T21:40:44.7181045Z 2022-03-17T21:40:44.7181321Z Generating XML reports... 2022-03-17T21:40:44.7260389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214029.xml 2022-03-17T21:40:47.4003662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:40:47.4015385Z 2022-03-17T21:40:47.4015703Z Running tests... 2022-03-17T21:40:47.4016235Z ---------------------------------------------------------------------- 2022-03-17T21:40:48.0130886Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75198 2022-03-17T21:40:48.0183401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75199 2022-03-17T21:40:48.0185465Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:40:49.1453781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:40:49.1455949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:49.1457259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:40:49.1458957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:40:49.1470848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:40:49.2465934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:40:58.9305631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:58.9307613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:58.9609498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:58.9610796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:40:58.9733900Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:58.9736108Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:40:58.9738906Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:40:58.9740822Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:41:01.6546350Z ok (14.253s) 2022-03-17T21:41:01.6546791Z 2022-03-17T21:41:01.6547796Z ---------------------------------------------------------------------- 2022-03-17T21:41:01.6548566Z Ran 1 test in 14.253s 2022-03-17T21:41:01.6548933Z 2022-03-17T21:41:01.6550184Z OK 2022-03-17T21:41:01.6550475Z 2022-03-17T21:41:01.6550781Z Generating XML reports... 2022-03-17T21:41:01.6626889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214047.xml 2022-03-17T21:41:04.3131224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:04.3140996Z 2022-03-17T21:41:04.3141457Z Running tests... 2022-03-17T21:41:04.3142533Z ---------------------------------------------------------------------- 2022-03-17T21:41:04.9158167Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75412 2022-03-17T21:41:04.9192914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75413 2022-03-17T21:41:04.9194232Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:06.0663870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:06.0664779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:06.0665362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:06.0666511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:06.0679476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:06.1681092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:06.5765442Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:06.5767039Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:06.8685755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:06.8688089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:06.8941506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:06.8943429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:06.9018576Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:41:06.9020917Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:41:06.9042738Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:41:06.9045149Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:41:08.7334498Z ok (4.418s) 2022-03-17T21:41:08.7334988Z 2022-03-17T21:41:08.7335982Z ---------------------------------------------------------------------- 2022-03-17T21:41:08.7336943Z Ran 1 test in 4.419s 2022-03-17T21:41:08.7337380Z 2022-03-17T21:41:08.7337557Z OK 2022-03-17T21:41:08.7337892Z 2022-03-17T21:41:08.7338205Z Generating XML reports... 2022-03-17T21:41:08.7414013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214104.xml 2022-03-17T21:41:11.3738601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:11.3750601Z 2022-03-17T21:41:11.3751196Z Running tests... 2022-03-17T21:41:11.3752428Z ---------------------------------------------------------------------- 2022-03-17T21:41:11.9731684Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75626 2022-03-17T21:41:11.9780161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75627 2022-03-17T21:41:11.9781550Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:13.1242839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:13.1243955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:13.1245712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:13.1345677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:13.1348889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:13.2268674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:14.8872015Z skip (3.511s) 2022-03-17T21:41:14.8873172Z 2022-03-17T21:41:14.8875645Z ---------------------------------------------------------------------- 2022-03-17T21:41:14.8876511Z Ran 1 test in 3.512s 2022-03-17T21:41:14.8876905Z 2022-03-17T21:41:14.8877178Z OK (skipped=1) 2022-03-17T21:41:14.8877564Z 2022-03-17T21:41:14.8877864Z Generating XML reports... 2022-03-17T21:41:14.8945499Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214111.xml 2022-03-17T21:41:17.4358534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:17.4368920Z 2022-03-17T21:41:17.4369230Z Running tests... 2022-03-17T21:41:17.4369804Z ---------------------------------------------------------------------- 2022-03-17T21:41:18.0678554Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75836 2022-03-17T21:41:18.0726226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75837 2022-03-17T21:41:18.0728193Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:19.2255087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:19.2256109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:19.2257140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:19.2356725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:19.2368609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:19.3371345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:21.1808150Z ok (3.743s) 2022-03-17T21:41:21.1808595Z 2022-03-17T21:41:21.1809502Z ---------------------------------------------------------------------- 2022-03-17T21:41:21.1810336Z Ran 1 test in 3.743s 2022-03-17T21:41:21.1810744Z 2022-03-17T21:41:21.1810987Z OK 2022-03-17T21:41:21.1811338Z 2022-03-17T21:41:21.1811655Z Generating XML reports... 2022-03-17T21:41:21.1889781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214117.xml 2022-03-17T21:41:23.8536926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:23.8547860Z 2022-03-17T21:41:23.8548394Z Running tests... 2022-03-17T21:41:23.8549602Z ---------------------------------------------------------------------- 2022-03-17T21:41:24.4519599Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76046 2022-03-17T21:41:24.4554650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76047 2022-03-17T21:41:24.4556077Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:25.6080669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:25.6081206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:25.6082164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:25.6082988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:25.6092216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:25.7100029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:35.5342373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:35.5343739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:35.6319657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:35.6321186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:38.3892298Z ok (14.534s) 2022-03-17T21:41:38.3896923Z 2022-03-17T21:41:38.3897984Z ---------------------------------------------------------------------- 2022-03-17T21:41:38.3898897Z Ran 1 test in 14.534s 2022-03-17T21:41:38.3899301Z 2022-03-17T21:41:38.3899542Z OK 2022-03-17T21:41:38.3899824Z 2022-03-17T21:41:38.3900132Z Generating XML reports... 2022-03-17T21:41:38.3976028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214123.xml 2022-03-17T21:41:40.9498845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:40.9508907Z 2022-03-17T21:41:40.9509571Z Running tests... 2022-03-17T21:41:40.9510781Z ---------------------------------------------------------------------- 2022-03-17T21:41:41.5520191Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76260 2022-03-17T21:41:41.5559836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76261 2022-03-17T21:41:41.5561167Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:42.7104018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:42.7105100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:42.7106919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:42.7206959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:42.7221605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:42.8218053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:45.0669718Z ok (4.115s) 2022-03-17T21:41:45.0670103Z 2022-03-17T21:41:45.0671050Z ---------------------------------------------------------------------- 2022-03-17T21:41:45.0671912Z Ran 1 test in 4.116s 2022-03-17T21:41:45.0672336Z 2022-03-17T21:41:45.0672510Z OK 2022-03-17T21:41:45.0672831Z 2022-03-17T21:41:45.0673134Z Generating XML reports... 2022-03-17T21:41:45.0744889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214140.xml 2022-03-17T21:41:47.6549171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:47.6560382Z 2022-03-17T21:41:47.6560685Z Running tests... 2022-03-17T21:41:47.6561172Z ---------------------------------------------------------------------- 2022-03-17T21:41:47.6579687Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:41:47.6580195Z 2022-03-17T21:41:47.6580701Z ---------------------------------------------------------------------- 2022-03-17T21:41:47.6581150Z Ran 1 test in 0.002s 2022-03-17T21:41:47.6581388Z 2022-03-17T21:41:47.6581542Z OK (skipped=1) 2022-03-17T21:41:47.6581760Z 2022-03-17T21:41:47.6581878Z Generating XML reports... 2022-03-17T21:41:47.6616786Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214147.xml 2022-03-17T21:41:50.1684211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:50.1692416Z 2022-03-17T21:41:50.1692880Z Running tests... 2022-03-17T21:41:50.1694353Z ---------------------------------------------------------------------- 2022-03-17T21:41:50.1709620Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T21:41:50.7709825Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76537 2022-03-17T21:41:50.7745292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76538 2022-03-17T21:41:50.7746891Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:51.9411343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:51.9413863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:51.9415266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:51.9417023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:51.9526626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:52.0436591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:53.7824388Z ok (3.613s) 2022-03-17T21:41:53.7824828Z 2022-03-17T21:41:53.7825798Z ---------------------------------------------------------------------- 2022-03-17T21:41:53.7826636Z Ran 1 test in 3.613s 2022-03-17T21:41:53.7827337Z 2022-03-17T21:41:53.7827593Z OK 2022-03-17T21:41:53.7836278Z 2022-03-17T21:41:53.7836995Z Generating XML reports... 2022-03-17T21:41:53.7904951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214150.xml 2022-03-17T21:41:56.3853325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:41:56.3860903Z 2022-03-17T21:41:56.3861098Z Running tests... 2022-03-17T21:41:56.3861808Z ---------------------------------------------------------------------- 2022-03-17T21:41:56.3884004Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:41:56.9886094Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76813 2022-03-17T21:41:56.9928713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76814 2022-03-17T21:41:56.9930081Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:41:58.1768575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:41:58.1770017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:41:58.1772109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:58.1869937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:41:58.1892345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:41:58.2879335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:41:58.3182770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:41:58.3183925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:42:00.1027339Z ok (3.716s) 2022-03-17T21:42:00.1027817Z 2022-03-17T21:42:00.1029028Z ---------------------------------------------------------------------- 2022-03-17T21:42:00.1029902Z Ran 1 test in 3.716s 2022-03-17T21:42:00.1030316Z 2022-03-17T21:42:00.1030554Z OK 2022-03-17T21:42:00.1030916Z 2022-03-17T21:42:00.1031189Z Generating XML reports... 2022-03-17T21:42:00.1103971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214156.xml 2022-03-17T21:42:02.6609088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:02.6619281Z 2022-03-17T21:42:02.6619748Z Running tests... 2022-03-17T21:42:02.6620861Z ---------------------------------------------------------------------- 2022-03-17T21:42:02.6642964Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:42:03.2630743Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77089 2022-03-17T21:42:03.2661890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77090 2022-03-17T21:42:03.2663333Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:04.4093160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:04.4093706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:04.4094480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:04.4095273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:04.4098997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:04.4102307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:04.4521202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:42:04.4521819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:42:06.1739872Z ok (3.512s) 2022-03-17T21:42:06.1740747Z 2022-03-17T21:42:06.1741778Z ---------------------------------------------------------------------- 2022-03-17T21:42:06.1743513Z Ran 1 test in 3.512s 2022-03-17T21:42:06.1743875Z 2022-03-17T21:42:06.1744091Z OK 2022-03-17T21:42:06.1744416Z 2022-03-17T21:42:06.1744736Z Generating XML reports... 2022-03-17T21:42:06.1821772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214202.xml 2022-03-17T21:42:08.7802518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:08.7813699Z 2022-03-17T21:42:08.7814305Z Running tests... 2022-03-17T21:42:08.7816014Z ---------------------------------------------------------------------- 2022-03-17T21:42:08.7829641Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:42:09.3806831Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77365 2022-03-17T21:42:09.3839844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77366 2022-03-17T21:42:09.3841483Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:10.4949532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:10.4950883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:10.4952548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:10.5053343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:10.5056929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:10.5967716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:12.1925953Z ok (3.411s) 2022-03-17T21:42:12.1926584Z 2022-03-17T21:42:12.1927739Z ---------------------------------------------------------------------- 2022-03-17T21:42:12.1928783Z Ran 1 test in 3.411s 2022-03-17T21:42:12.1929545Z 2022-03-17T21:42:12.1929808Z OK 2022-03-17T21:42:12.1930136Z 2022-03-17T21:42:12.1930435Z Generating XML reports... 2022-03-17T21:42:12.2010675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214208.xml 2022-03-17T21:42:14.8100224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:14.8112184Z 2022-03-17T21:42:14.8112608Z Running tests... 2022-03-17T21:42:14.8113650Z ---------------------------------------------------------------------- 2022-03-17T21:42:15.4074350Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77641 2022-03-17T21:42:15.4125928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77642 2022-03-17T21:42:15.4127241Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:16.4926988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:16.4927477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:16.4928294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:16.5030958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:16.5031578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:16.5939981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:18.6228188Z ok (3.811s) 2022-03-17T21:42:18.6228813Z 2022-03-17T21:42:18.6230154Z ---------------------------------------------------------------------- 2022-03-17T21:42:18.6231009Z Ran 1 test in 3.811s 2022-03-17T21:42:18.6231407Z 2022-03-17T21:42:18.6231595Z OK 2022-03-17T21:42:18.6231912Z 2022-03-17T21:42:18.6232220Z Generating XML reports... 2022-03-17T21:42:18.6303276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214214.xml 2022-03-17T21:42:21.1841071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:21.1854000Z 2022-03-17T21:42:21.1854534Z Running tests... 2022-03-17T21:42:21.1855145Z ---------------------------------------------------------------------- 2022-03-17T21:42:21.8207551Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77857 2022-03-17T21:42:21.8260370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77858 2022-03-17T21:42:21.8261499Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:22.9724028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:22.9724628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:22.9725645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:22.9726488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:22.9729906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:22.9731621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:25.1370692Z ok (3.951s) 2022-03-17T21:42:25.1371399Z 2022-03-17T21:42:25.1372544Z ---------------------------------------------------------------------- 2022-03-17T21:42:25.1373722Z Ran 1 test in 3.952s 2022-03-17T21:42:25.1374285Z 2022-03-17T21:42:25.1374596Z OK 2022-03-17T21:42:25.1377769Z 2022-03-17T21:42:25.1378123Z Generating XML reports... 2022-03-17T21:42:25.1461548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214221.xml 2022-03-17T21:42:27.7371268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:27.7383408Z 2022-03-17T21:42:27.7383723Z Running tests... 2022-03-17T21:42:27.7384166Z ---------------------------------------------------------------------- 2022-03-17T21:42:28.3694732Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78073 2022-03-17T21:42:28.3741794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78074 2022-03-17T21:42:28.3743838Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:29.5223480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:29.5224556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:29.5226391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:29.5325060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:29.5328654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:29.6249656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:29.6447988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:42:29.6449043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:42:29.6450798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:42:29.6549709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:42:31.4824749Z ok (3.744s) 2022-03-17T21:42:31.4825143Z 2022-03-17T21:42:31.4826252Z ---------------------------------------------------------------------- 2022-03-17T21:42:31.4827106Z Ran 1 test in 3.744s 2022-03-17T21:42:31.4827501Z 2022-03-17T21:42:31.4827670Z OK 2022-03-17T21:42:31.4828005Z 2022-03-17T21:42:31.4828312Z Generating XML reports... 2022-03-17T21:42:31.4901953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214227.xml 2022-03-17T21:42:34.2062929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:34.2075707Z 2022-03-17T21:42:34.2076281Z Running tests... 2022-03-17T21:42:34.2077543Z ---------------------------------------------------------------------- 2022-03-17T21:42:34.8195339Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78295 2022-03-17T21:42:34.8255776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78296 2022-03-17T21:42:34.8257436Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:35.9906101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:35.9907631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:35.9909812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:35.9911689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:35.9923797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:35.9925399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:37.6354259Z skip (3.427s) 2022-03-17T21:42:37.6354944Z 2022-03-17T21:42:37.6356094Z ---------------------------------------------------------------------- 2022-03-17T21:42:37.6356977Z Ran 1 test in 3.428s 2022-03-17T21:42:37.6357356Z 2022-03-17T21:42:37.6357662Z OK (skipped=1) 2022-03-17T21:42:37.6358059Z 2022-03-17T21:42:37.6358397Z Generating XML reports... 2022-03-17T21:42:37.6426189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214234.xml 2022-03-17T21:42:40.2872715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:40.2885851Z 2022-03-17T21:42:40.2886183Z Running tests... 2022-03-17T21:42:40.2886622Z ---------------------------------------------------------------------- 2022-03-17T21:42:40.9031703Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78505 2022-03-17T21:42:40.9088737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78506 2022-03-17T21:42:40.9090772Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:42.0655283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:42.0657815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:42.0659014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:42.0660384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:42.0763327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:42.1677438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:43.9182546Z ok (3.629s) 2022-03-17T21:42:43.9182965Z 2022-03-17T21:42:43.9183979Z ---------------------------------------------------------------------- 2022-03-17T21:42:43.9184969Z Ran 1 test in 3.630s 2022-03-17T21:42:43.9185380Z 2022-03-17T21:42:43.9185555Z OK 2022-03-17T21:42:43.9185911Z 2022-03-17T21:42:43.9188114Z Generating XML reports... 2022-03-17T21:42:43.9266432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214240.xml 2022-03-17T21:42:46.5832367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:46.5842234Z 2022-03-17T21:42:46.5842501Z Running tests... 2022-03-17T21:42:46.5842987Z ---------------------------------------------------------------------- 2022-03-17T21:42:47.1826218Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78721 2022-03-17T21:42:47.1860320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78722 2022-03-17T21:42:47.1861710Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:48.3440026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:48.3440541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:48.3441352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:48.3442300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:48.3445733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:48.3446295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:49.9951344Z ok (3.410s) 2022-03-17T21:42:49.9951751Z 2022-03-17T21:42:49.9952679Z ---------------------------------------------------------------------- 2022-03-17T21:42:49.9953543Z Ran 1 test in 3.411s 2022-03-17T21:42:49.9953955Z 2022-03-17T21:42:49.9954206Z OK 2022-03-17T21:42:49.9954533Z 2022-03-17T21:42:49.9954800Z Generating XML reports... 2022-03-17T21:42:50.0021835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214246.xml 2022-03-17T21:42:52.5915833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:52.5926964Z 2022-03-17T21:42:52.5927228Z Running tests... 2022-03-17T21:42:52.5928334Z ---------------------------------------------------------------------- 2022-03-17T21:42:53.1885464Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78937 2022-03-17T21:42:53.1956630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78938 2022-03-17T21:42:53.1960545Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:42:54.3333041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:42:54.3333749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:42:54.3334556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:54.3335393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:42:54.3339030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:42:54.3339671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:42:56.0063295Z ok (3.413s) 2022-03-17T21:42:56.0063787Z 2022-03-17T21:42:56.0065115Z ---------------------------------------------------------------------- 2022-03-17T21:42:56.0066174Z Ran 1 test in 3.413s 2022-03-17T21:42:56.0066678Z 2022-03-17T21:42:56.0066919Z OK 2022-03-17T21:42:56.0067313Z 2022-03-17T21:42:56.0067695Z Generating XML reports... 2022-03-17T21:42:56.0144872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214252.xml 2022-03-17T21:42:58.7478327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:42:58.7491045Z 2022-03-17T21:42:58.7491747Z Running tests... 2022-03-17T21:42:58.7493240Z ---------------------------------------------------------------------- 2022-03-17T21:42:58.7510604Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:42:58.7511435Z 2022-03-17T21:42:58.7512470Z ---------------------------------------------------------------------- 2022-03-17T21:42:58.7513479Z Ran 1 test in 0.002s 2022-03-17T21:42:58.7513899Z 2022-03-17T21:42:58.7514169Z OK (skipped=1) 2022-03-17T21:42:58.7514559Z 2022-03-17T21:42:58.7514823Z Generating XML reports... 2022-03-17T21:42:58.7551750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214258.xml 2022-03-17T21:43:01.3723805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:01.3737518Z 2022-03-17T21:43:01.3737785Z Running tests... 2022-03-17T21:43:01.3738253Z ---------------------------------------------------------------------- 2022-03-17T21:43:01.3753507Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:43:01.3753774Z 2022-03-17T21:43:01.3754097Z ---------------------------------------------------------------------- 2022-03-17T21:43:01.3754435Z Ran 1 test in 0.002s 2022-03-17T21:43:01.3754595Z 2022-03-17T21:43:01.3754719Z OK (skipped=1) 2022-03-17T21:43:01.3754901Z 2022-03-17T21:43:01.3755036Z Generating XML reports... 2022-03-17T21:43:01.3797557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214301.xml 2022-03-17T21:43:03.9895123Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:03.9910512Z 2022-03-17T21:43:03.9910779Z Running tests... 2022-03-17T21:43:04.5871250Z ---------------------------------------------------------------------- 2022-03-17T21:43:04.5872727Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79287 2022-03-17T21:43:04.5903685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79288 2022-03-17T21:43:04.5905066Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:05.7157652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:05.7158767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:05.7161311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:05.7260126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:05.7261917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:05.8177880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:05.8388074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:43:05.8389986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:05.8391309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:43:05.8393041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:07.8004696Z ok (3.809s) 2022-03-17T21:43:07.8005239Z 2022-03-17T21:43:07.8006266Z ---------------------------------------------------------------------- 2022-03-17T21:43:07.8007171Z Ran 1 test in 3.810s 2022-03-17T21:43:07.8007577Z 2022-03-17T21:43:07.8007746Z OK 2022-03-17T21:43:07.8008064Z 2022-03-17T21:43:07.8008378Z Generating XML reports... 2022-03-17T21:43:07.8077858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214303.xml 2022-03-17T21:43:10.6300328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:10.6314938Z 2022-03-17T21:43:10.6315334Z Running tests... 2022-03-17T21:43:11.2543708Z ---------------------------------------------------------------------- 2022-03-17T21:43:11.2545160Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79509 2022-03-17T21:43:11.2593115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79510 2022-03-17T21:43:11.2595271Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:12.4220445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:12.4222401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:12.4224577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:12.4324497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:12.4339272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:12.5333859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:14.2699719Z skip (3.638s) 2022-03-17T21:43:14.2700204Z 2022-03-17T21:43:14.2701229Z ---------------------------------------------------------------------- 2022-03-17T21:43:14.2702025Z Ran 1 test in 3.638s 2022-03-17T21:43:14.2702420Z 2022-03-17T21:43:14.2702672Z OK (skipped=1) 2022-03-17T21:43:14.2703043Z 2022-03-17T21:43:14.2703296Z Generating XML reports... 2022-03-17T21:43:14.2782092Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214310.xml 2022-03-17T21:43:16.9613004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:16.9627797Z 2022-03-17T21:43:16.9628090Z Running tests... 2022-03-17T21:43:16.9628579Z ---------------------------------------------------------------------- 2022-03-17T21:43:16.9645531Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:43:16.9646118Z 2022-03-17T21:43:16.9646442Z ---------------------------------------------------------------------- 2022-03-17T21:43:16.9646857Z Ran 1 test in 0.002s 2022-03-17T21:43:16.9647042Z 2022-03-17T21:43:16.9647159Z OK (skipped=1) 2022-03-17T21:43:16.9647311Z 2022-03-17T21:43:16.9647450Z Generating XML reports... 2022-03-17T21:43:16.9685230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214316.xml 2022-03-17T21:43:19.7126616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:19.7139826Z 2022-03-17T21:43:19.7140684Z Running tests... 2022-03-17T21:43:19.7141994Z ---------------------------------------------------------------------- 2022-03-17T21:43:19.7160967Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:43:19.7162342Z 2022-03-17T21:43:19.7163252Z ---------------------------------------------------------------------- 2022-03-17T21:43:19.7164288Z Ran 1 test in 0.002s 2022-03-17T21:43:19.7164638Z 2022-03-17T21:43:19.7165001Z OK (skipped=1) 2022-03-17T21:43:19.7165383Z 2022-03-17T21:43:19.7165699Z Generating XML reports... 2022-03-17T21:43:19.7201217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214319.xml 2022-03-17T21:43:22.2258270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:22.2273880Z 2022-03-17T21:43:22.2274137Z Running tests... 2022-03-17T21:43:22.2274640Z ---------------------------------------------------------------------- 2022-03-17T21:43:22.8224636Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79853 2022-03-17T21:43:22.8259407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79854 2022-03-17T21:43:22.8260663Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:24.0008457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:24.0008996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:24.0010130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:24.0010962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:24.0022319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:24.0024532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:25.7355745Z ok (3.508s) 2022-03-17T21:43:25.7356336Z 2022-03-17T21:43:25.7357575Z ---------------------------------------------------------------------- 2022-03-17T21:43:25.7358571Z Ran 1 test in 3.508s 2022-03-17T21:43:25.7359100Z 2022-03-17T21:43:25.7359329Z OK 2022-03-17T21:43:25.7359660Z 2022-03-17T21:43:25.7359934Z Generating XML reports... 2022-03-17T21:43:25.7433667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214322.xml 2022-03-17T21:43:28.2915005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:28.2928461Z 2022-03-17T21:43:28.2928750Z Running tests... 2022-03-17T21:43:28.2929271Z ---------------------------------------------------------------------- 2022-03-17T21:43:28.8912754Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80063 2022-03-17T21:43:28.8946905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80064 2022-03-17T21:43:28.8948682Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:30.0485360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:30.0502350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:30.0504933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:30.0589399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:30.0603559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:30.1526862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:30.1948480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:43:30.1950153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:43:30.1952278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:30.1954243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:30.2234165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:43:30.2235710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:43:30.2238281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:43:30.2240852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:43:30.2474322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T21:43:30.2475969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T21:43:30.2478124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:43:30.2480499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:43:32.0031893Z ok (3.710s) 2022-03-17T21:43:32.0032361Z 2022-03-17T21:43:32.0033322Z ---------------------------------------------------------------------- 2022-03-17T21:43:32.0034215Z Ran 1 test in 3.710s 2022-03-17T21:43:32.0034627Z 2022-03-17T21:43:32.0034875Z OK 2022-03-17T21:43:32.0035157Z 2022-03-17T21:43:32.0035484Z Generating XML reports... 2022-03-17T21:43:32.0108910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214328.xml 2022-03-17T21:43:34.6098192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:34.6110517Z 2022-03-17T21:43:34.6110672Z Running tests... 2022-03-17T21:43:34.6112976Z ---------------------------------------------------------------------- 2022-03-17T21:43:35.2371533Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80297 2022-03-17T21:43:35.2423098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80298 2022-03-17T21:43:35.2424544Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:36.3843960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:36.3844477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:36.3845904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:36.3846719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:36.3953627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:36.4858422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:36.4966379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:43:36.4967643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:43:36.4969663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:36.4971504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:38.3510969Z ok (3.739s) 2022-03-17T21:43:38.3511435Z 2022-03-17T21:43:38.3512380Z ---------------------------------------------------------------------- 2022-03-17T21:43:38.3513377Z Ran 1 test in 3.740s 2022-03-17T21:43:38.3513737Z 2022-03-17T21:43:38.3513945Z OK 2022-03-17T21:43:38.3514264Z 2022-03-17T21:43:38.3514589Z Generating XML reports... 2022-03-17T21:43:38.3586565Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214334.xml 2022-03-17T21:43:41.0179188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:41.0190177Z 2022-03-17T21:43:41.0190673Z Running tests... 2022-03-17T21:43:41.6170006Z ---------------------------------------------------------------------- 2022-03-17T21:43:41.6171428Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80519 2022-03-17T21:43:41.6207398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80520 2022-03-17T21:43:41.6208766Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:42.7583385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:42.7584887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:42.7586875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:42.7685810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:42.7703514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:42.8696062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:42.8901284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:43:42.8902894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:43:42.8904928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:42.9005894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:44.8310166Z ok (3.811s) 2022-03-17T21:43:44.8310882Z 2022-03-17T21:43:44.8311948Z ---------------------------------------------------------------------- 2022-03-17T21:43:44.8313005Z Ran 1 test in 3.812s 2022-03-17T21:43:44.8313427Z 2022-03-17T21:43:44.8314192Z OK 2022-03-17T21:43:44.8314526Z 2022-03-17T21:43:44.8314846Z Generating XML reports... 2022-03-17T21:43:44.8393544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214341.xml 2022-03-17T21:43:47.6463216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:47.6477466Z 2022-03-17T21:43:47.6477872Z Running tests... 2022-03-17T21:43:47.6478337Z ---------------------------------------------------------------------- 2022-03-17T21:43:48.2466031Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80741 2022-03-17T21:43:48.2500741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80742 2022-03-17T21:43:48.2502242Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:49.3643095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:49.3643723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:49.3644726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:49.3645441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:49.3650021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:49.3650601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:49.3857256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:43:49.3857833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:43:49.3858670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:49.3859363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:51.2580514Z ok (3.610s) 2022-03-17T21:43:51.2581194Z 2022-03-17T21:43:51.2582395Z ---------------------------------------------------------------------- 2022-03-17T21:43:51.2583964Z Ran 1 test in 3.610s 2022-03-17T21:43:51.2584551Z 2022-03-17T21:43:51.2584800Z OK 2022-03-17T21:43:51.2585149Z 2022-03-17T21:43:51.2585460Z Generating XML reports... 2022-03-17T21:43:51.2660896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214347.xml 2022-03-17T21:43:53.8156342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:43:53.8170269Z 2022-03-17T21:43:53.8171063Z Running tests... 2022-03-17T21:43:53.8172502Z ---------------------------------------------------------------------- 2022-03-17T21:43:54.4174430Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80963 2022-03-17T21:43:54.4221127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80964 2022-03-17T21:43:54.4222762Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:43:55.5314939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:43:55.5319440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:43:55.5320214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:55.5417300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:43:55.5420733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:43:55.6337085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:43:55.6439921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:43:55.6440584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:43:55.6441308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:55.6541542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:43:58.2336476Z ok (4.416s) 2022-03-17T21:43:58.2336898Z 2022-03-17T21:43:58.2337856Z ---------------------------------------------------------------------- 2022-03-17T21:43:58.2338743Z Ran 1 test in 4.417s 2022-03-17T21:43:58.2339094Z 2022-03-17T21:43:58.2339312Z OK 2022-03-17T21:43:58.2339637Z 2022-03-17T21:43:58.2339947Z Generating XML reports... 2022-03-17T21:43:58.2418141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214353.xml 2022-03-17T21:44:00.7894361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:00.7908607Z 2022-03-17T21:44:00.7909009Z Running tests... 2022-03-17T21:44:00.7909453Z ---------------------------------------------------------------------- 2022-03-17T21:44:01.3890494Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81185 2022-03-17T21:44:01.3937250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81186 2022-03-17T21:44:01.3938544Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:02.5250507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:02.5251309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:02.5252157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:02.5252932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:02.5257220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:02.5259107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:04.2029074Z skip (3.412s) 2022-03-17T21:44:04.2029664Z 2022-03-17T21:44:04.2030749Z ---------------------------------------------------------------------- 2022-03-17T21:44:04.2031793Z Ran 1 test in 3.412s 2022-03-17T21:44:04.2032299Z 2022-03-17T21:44:04.2032569Z OK (skipped=1) 2022-03-17T21:44:04.2032978Z 2022-03-17T21:44:04.2033270Z Generating XML reports... 2022-03-17T21:44:04.2101734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214400.xml 2022-03-17T21:44:06.7628947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:06.7641422Z 2022-03-17T21:44:06.7642744Z Running tests... 2022-03-17T21:44:06.7644250Z ---------------------------------------------------------------------- 2022-03-17T21:44:07.4020754Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81395 2022-03-17T21:44:07.4065319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81396 2022-03-17T21:44:07.4067502Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:08.5376460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:08.5377789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:08.5379682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:08.5381539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:08.5383113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:08.5387039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:10.2153146Z skip (3.450s) 2022-03-17T21:44:10.2153668Z 2022-03-17T21:44:10.2154618Z ---------------------------------------------------------------------- 2022-03-17T21:44:10.2155476Z Ran 1 test in 3.451s 2022-03-17T21:44:10.2156180Z 2022-03-17T21:44:10.2156392Z OK (skipped=1) 2022-03-17T21:44:10.2159421Z 2022-03-17T21:44:10.2160198Z Generating XML reports... 2022-03-17T21:44:10.2230877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214406.xml 2022-03-17T21:44:12.9042810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:12.9054601Z 2022-03-17T21:44:12.9055243Z Running tests... 2022-03-17T21:44:12.9056289Z ---------------------------------------------------------------------- 2022-03-17T21:44:13.5276535Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81605 2022-03-17T21:44:13.5354595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81606 2022-03-17T21:44:13.5355409Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:14.7029390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:14.7031605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:14.7034003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:14.7134353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:14.7149198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:14.8143414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:17.1461436Z skip (4.240s) 2022-03-17T21:44:17.1462193Z 2022-03-17T21:44:17.1463309Z ---------------------------------------------------------------------- 2022-03-17T21:44:17.1464556Z Ran 1 test in 4.240s 2022-03-17T21:44:17.1465157Z 2022-03-17T21:44:17.1465405Z OK (skipped=1) 2022-03-17T21:44:17.1465796Z 2022-03-17T21:44:17.1466112Z Generating XML reports... 2022-03-17T21:44:17.1538426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214412.xml 2022-03-17T21:44:19.8789144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:19.8802413Z 2022-03-17T21:44:19.8802744Z Running tests... 2022-03-17T21:44:19.8803742Z ---------------------------------------------------------------------- 2022-03-17T21:44:20.4794625Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81815 2022-03-17T21:44:20.4831811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81816 2022-03-17T21:44:20.4833190Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:21.6435895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:21.6437226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:21.6439074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:21.6537864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:21.6555235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:21.7548105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:23.3926232Z skip (3.512s) 2022-03-17T21:44:23.3926675Z 2022-03-17T21:44:23.3927518Z ---------------------------------------------------------------------- 2022-03-17T21:44:23.3928319Z Ran 1 test in 3.512s 2022-03-17T21:44:23.3928719Z 2022-03-17T21:44:23.3928985Z OK (skipped=1) 2022-03-17T21:44:23.3929358Z 2022-03-17T21:44:23.3929608Z Generating XML reports... 2022-03-17T21:44:23.4009933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214419.xml 2022-03-17T21:44:25.9935355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:25.9948046Z 2022-03-17T21:44:25.9948420Z Running tests... 2022-03-17T21:44:25.9949067Z ---------------------------------------------------------------------- 2022-03-17T21:44:26.6122848Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82025 2022-03-17T21:44:26.6175039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82026 2022-03-17T21:44:26.6176417Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:27.7841436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:27.7843656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:27.7845636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:27.7847461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:27.7949118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:27.8858501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:29.7257354Z ok (3.730s) 2022-03-17T21:44:29.7258062Z 2022-03-17T21:44:29.7259694Z ---------------------------------------------------------------------- 2022-03-17T21:44:29.7260618Z Ran 1 test in 3.731s 2022-03-17T21:44:29.7261043Z 2022-03-17T21:44:29.7261214Z OK 2022-03-17T21:44:29.7261549Z 2022-03-17T21:44:29.7261867Z Generating XML reports... 2022-03-17T21:44:29.7344197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214425.xml 2022-03-17T21:44:32.3825303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:32.3843010Z 2022-03-17T21:44:32.3843331Z Running tests... 2022-03-17T21:44:32.3843877Z ---------------------------------------------------------------------- 2022-03-17T21:44:33.0088773Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82241 2022-03-17T21:44:33.0134901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82242 2022-03-17T21:44:33.0136410Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:34.1714049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:34.1715472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:34.1717532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:34.1812908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:34.1815865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:34.2738072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:35.9229971Z ok (3.538s) 2022-03-17T21:44:35.9230537Z 2022-03-17T21:44:35.9231801Z ---------------------------------------------------------------------- 2022-03-17T21:44:35.9232769Z Ran 1 test in 3.539s 2022-03-17T21:44:35.9233261Z 2022-03-17T21:44:35.9233536Z OK 2022-03-17T21:44:35.9234023Z 2022-03-17T21:44:35.9234367Z Generating XML reports... 2022-03-17T21:44:35.9307980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214432.xml 2022-03-17T21:44:38.5725682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:38.5738574Z 2022-03-17T21:44:38.5738914Z Running tests... 2022-03-17T21:44:38.5739362Z ---------------------------------------------------------------------- 2022-03-17T21:44:39.1721012Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82451 2022-03-17T21:44:39.1801712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82452 2022-03-17T21:44:39.1803530Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:40.2990608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:40.2992625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:40.2994024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:40.2995802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:40.2997145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:40.3997040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:42.1893063Z ok (3.615s) 2022-03-17T21:44:42.1893546Z 2022-03-17T21:44:42.1894707Z ---------------------------------------------------------------------- 2022-03-17T21:44:42.1895733Z Ran 1 test in 3.615s 2022-03-17T21:44:42.1896225Z 2022-03-17T21:44:42.1896434Z OK 2022-03-17T21:44:42.1896852Z 2022-03-17T21:44:42.1899530Z Generating XML reports... 2022-03-17T21:44:42.1976840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214438.xml 2022-03-17T21:44:45.0566221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:45.0579063Z 2022-03-17T21:44:45.0581440Z Running tests... 2022-03-17T21:44:45.0582147Z ---------------------------------------------------------------------- 2022-03-17T21:44:45.6574564Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82667 2022-03-17T21:44:45.6632229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82668 2022-03-17T21:44:45.6634017Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:46.8331209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:46.8332401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:46.8333930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:46.8335341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:46.8336409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:46.8337357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:48.5718079Z ok (3.513s) 2022-03-17T21:44:48.5718511Z 2022-03-17T21:44:48.5719685Z ---------------------------------------------------------------------- 2022-03-17T21:44:48.5720616Z Ran 1 test in 3.514s 2022-03-17T21:44:48.5721129Z 2022-03-17T21:44:48.5721421Z OK 2022-03-17T21:44:48.5722729Z 2022-03-17T21:44:48.5723276Z Generating XML reports... 2022-03-17T21:44:48.5803094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214445.xml 2022-03-17T21:44:51.1381072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:51.1393174Z 2022-03-17T21:44:51.1393481Z Running tests... 2022-03-17T21:44:51.1394230Z ---------------------------------------------------------------------- 2022-03-17T21:44:51.7370858Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82883 2022-03-17T21:44:51.7419718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82884 2022-03-17T21:44:51.7421447Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:52.8880403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:52.8881410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:52.8882003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:52.8882624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:52.8893645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:44:52.9891047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:54.7539659Z ok (3.614s) 2022-03-17T21:44:54.7540318Z 2022-03-17T21:44:54.7541461Z ---------------------------------------------------------------------- 2022-03-17T21:44:54.7542514Z Ran 1 test in 3.615s 2022-03-17T21:44:54.7543004Z 2022-03-17T21:44:54.7543240Z OK 2022-03-17T21:44:54.7543636Z 2022-03-17T21:44:54.7544040Z Generating XML reports... 2022-03-17T21:44:54.7619318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214451.xml 2022-03-17T21:44:57.3119976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:44:57.3130416Z 2022-03-17T21:44:57.3131106Z Running tests... 2022-03-17T21:44:57.3132118Z ---------------------------------------------------------------------- 2022-03-17T21:44:57.9142265Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83099 2022-03-17T21:44:57.9195740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83100 2022-03-17T21:44:57.9197184Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:44:59.0824104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:44:59.0825994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:59.0827276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:44:59.0828821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:44:59.0830140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:44:59.0833985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:00.8289411Z ok (3.515s) 2022-03-17T21:45:00.8289848Z 2022-03-17T21:45:00.8290855Z ---------------------------------------------------------------------- 2022-03-17T21:45:00.8291717Z Ran 1 test in 3.516s 2022-03-17T21:45:00.8292093Z 2022-03-17T21:45:00.8292312Z OK 2022-03-17T21:45:00.8292632Z 2022-03-17T21:45:00.8292947Z Generating XML reports... 2022-03-17T21:45:00.8370882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214457.xml 2022-03-17T21:45:03.3819006Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:03.3829663Z 2022-03-17T21:45:03.3829946Z Running tests... 2022-03-17T21:45:03.3830445Z ---------------------------------------------------------------------- 2022-03-17T21:45:03.9808567Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83309 2022-03-17T21:45:03.9855934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83310 2022-03-17T21:45:03.9857781Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:05.0719907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:05.0739334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:05.0740957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:05.0822937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:05.0827323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:05.1751085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:05.1946521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:45:05.1947728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:45:05.1949515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:05.1951484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:07.3955793Z ok (4.012s) 2022-03-17T21:45:07.3956228Z 2022-03-17T21:45:07.3957396Z ---------------------------------------------------------------------- 2022-03-17T21:45:07.3958867Z Ran 1 test in 4.012s 2022-03-17T21:45:07.3959260Z 2022-03-17T21:45:07.3959480Z OK 2022-03-17T21:45:07.3959817Z 2022-03-17T21:45:07.3960126Z Generating XML reports... 2022-03-17T21:45:07.4028733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214503.xml 2022-03-17T21:45:09.9658824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:09.9669343Z 2022-03-17T21:45:09.9669889Z Running tests... 2022-03-17T21:45:09.9670438Z ---------------------------------------------------------------------- 2022-03-17T21:45:10.5673171Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83531 2022-03-17T21:45:10.5709265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83532 2022-03-17T21:45:10.5710664Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:11.7330911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:11.7331913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:11.7333008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:11.7335618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:11.7441960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:11.8359673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:11.8570347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:45:11.8571992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:45:11.8573997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:11.8672196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:13.6785356Z ok (3.711s) 2022-03-17T21:45:13.6786074Z 2022-03-17T21:45:13.6788057Z ---------------------------------------------------------------------- 2022-03-17T21:45:13.6789239Z Ran 1 test in 3.711s 2022-03-17T21:45:13.6840996Z 2022-03-17T21:45:13.6841240Z OK 2022-03-17T21:45:13.6841587Z 2022-03-17T21:45:13.6842033Z Generating XML reports... 2022-03-17T21:45:13.6860810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214509.xml 2022-03-17T21:45:16.3022392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:16.3032864Z 2022-03-17T21:45:16.3033194Z Running tests... 2022-03-17T21:45:16.3033631Z ---------------------------------------------------------------------- 2022-03-17T21:45:16.9006477Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83753 2022-03-17T21:45:16.9045709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83754 2022-03-17T21:45:16.9047038Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:18.0669336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:18.0671398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:18.0672835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:18.0675167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:18.0777930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:18.1689289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:18.1892295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:45:18.1893639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:45:18.1895581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:18.1993654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:20.3150845Z ok (4.011s) 2022-03-17T21:45:20.3151330Z 2022-03-17T21:45:20.3152415Z ---------------------------------------------------------------------- 2022-03-17T21:45:20.3153272Z Ran 1 test in 4.012s 2022-03-17T21:45:20.3153626Z 2022-03-17T21:45:20.3153843Z OK 2022-03-17T21:45:20.3154181Z 2022-03-17T21:45:20.3154492Z Generating XML reports... 2022-03-17T21:45:20.3226299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214516.xml 2022-03-17T21:45:22.8800932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:22.8817059Z 2022-03-17T21:45:22.8817392Z Running tests... 2022-03-17T21:45:22.8817837Z ---------------------------------------------------------------------- 2022-03-17T21:45:23.5308750Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83975 2022-03-17T21:45:23.5344827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83976 2022-03-17T21:45:23.5346117Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:24.6712754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:24.6714079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:24.6716604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:24.6718525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:24.6723912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:24.7725097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:24.7841141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:45:24.7842702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:45:24.7844564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:24.7947279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:45:26.5443055Z ok (3.662s) 2022-03-17T21:45:26.5443826Z 2022-03-17T21:45:26.5444870Z ---------------------------------------------------------------------- 2022-03-17T21:45:26.5445807Z Ran 1 test in 3.663s 2022-03-17T21:45:26.5446159Z 2022-03-17T21:45:26.5446409Z OK 2022-03-17T21:45:26.5446746Z 2022-03-17T21:45:26.5447065Z Generating XML reports... 2022-03-17T21:45:26.5534205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214522.xml 2022-03-17T21:45:29.4966839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:29.4980372Z 2022-03-17T21:45:29.4981028Z Running tests... 2022-03-17T21:45:29.4982077Z ---------------------------------------------------------------------- 2022-03-17T21:45:30.0969199Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84197 2022-03-17T21:45:30.1006489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84198 2022-03-17T21:45:30.1007940Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:31.2468703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:31.2470745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:31.2472104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:31.2473780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:31.2583389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:31.3492643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:33.2089021Z skip (3.710s) 2022-03-17T21:45:33.2089469Z 2022-03-17T21:45:33.2091711Z ---------------------------------------------------------------------- 2022-03-17T21:45:33.2092570Z Ran 1 test in 3.711s 2022-03-17T21:45:33.2092969Z 2022-03-17T21:45:33.2093229Z OK (skipped=1) 2022-03-17T21:45:33.2093611Z 2022-03-17T21:45:33.2093861Z Generating XML reports... 2022-03-17T21:45:33.2170209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214529.xml 2022-03-17T21:45:35.8820969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:35.8835394Z 2022-03-17T21:45:35.8835694Z Running tests... 2022-03-17T21:45:35.8836163Z ---------------------------------------------------------------------- 2022-03-17T21:45:36.4809781Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84407 2022-03-17T21:45:36.4866015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84408 2022-03-17T21:45:36.4867175Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:37.6298079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:37.6298569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:37.6299339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:37.6300161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:37.6407577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:37.7311630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:39.3944886Z skip (3.510s) 2022-03-17T21:45:39.3945555Z 2022-03-17T21:45:39.3946554Z ---------------------------------------------------------------------- 2022-03-17T21:45:39.3947471Z Ran 1 test in 3.511s 2022-03-17T21:45:39.3947819Z 2022-03-17T21:45:39.3948071Z OK (skipped=1) 2022-03-17T21:45:39.3948436Z 2022-03-17T21:45:39.3948709Z Generating XML reports... 2022-03-17T21:45:39.4021861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214535.xml 2022-03-17T21:45:42.0731592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:42.0742166Z 2022-03-17T21:45:42.0742522Z Running tests... 2022-03-17T21:45:42.0742997Z ---------------------------------------------------------------------- 2022-03-17T21:45:42.6707993Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84617 2022-03-17T21:45:42.6755140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84618 2022-03-17T21:45:42.6756314Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:43.8265594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:43.8266930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:43.8268915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:43.8270715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:43.8279937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:43.8281182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:45.6840655Z skip (3.609s) 2022-03-17T21:45:45.6841036Z 2022-03-17T21:45:45.6843632Z ---------------------------------------------------------------------- 2022-03-17T21:45:45.6844525Z Ran 1 test in 3.610s 2022-03-17T21:45:45.6844934Z 2022-03-17T21:45:45.6845149Z OK (skipped=1) 2022-03-17T21:45:45.6845525Z 2022-03-17T21:45:45.6845847Z Generating XML reports... 2022-03-17T21:45:45.6924561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214542.xml 2022-03-17T21:45:48.2778307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:48.2791253Z 2022-03-17T21:45:48.2842233Z Running tests... 2022-03-17T21:45:48.2842744Z ---------------------------------------------------------------------- 2022-03-17T21:45:48.8890237Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84827 2022-03-17T21:45:48.8972198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84828 2022-03-17T21:45:48.8973589Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:50.0664639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:50.0665161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:50.0665969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:50.0666811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:50.0774311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:50.1677307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:52.0055382Z skip (3.726s) 2022-03-17T21:45:52.0055855Z 2022-03-17T21:45:52.0058527Z ---------------------------------------------------------------------- 2022-03-17T21:45:52.0060003Z Ran 1 test in 3.726s 2022-03-17T21:45:52.0060410Z 2022-03-17T21:45:52.0060777Z OK (skipped=1) 2022-03-17T21:45:52.0061895Z 2022-03-17T21:45:52.0062156Z Generating XML reports... 2022-03-17T21:45:52.0141614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214548.xml 2022-03-17T21:45:54.9856683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:45:54.9868336Z 2022-03-17T21:45:54.9868684Z Running tests... 2022-03-17T21:45:55.5855168Z ---------------------------------------------------------------------- 2022-03-17T21:45:55.5856645Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85037 2022-03-17T21:45:55.5900624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85038 2022-03-17T21:45:55.5903765Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:45:56.7567454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:45:56.7569636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:56.7571045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:45:56.7572832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:45:56.7677956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:45:56.8589377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:45:58.6983866Z ok (3.711s) 2022-03-17T21:45:58.6984247Z 2022-03-17T21:45:58.6985116Z ---------------------------------------------------------------------- 2022-03-17T21:45:58.6985951Z Ran 1 test in 3.712s 2022-03-17T21:45:58.6986352Z 2022-03-17T21:45:58.6986592Z OK 2022-03-17T21:45:58.6986927Z 2022-03-17T21:45:58.6987250Z Generating XML reports... 2022-03-17T21:45:58.7063580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214554.xml 2022-03-17T21:46:01.3392772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:01.3407016Z 2022-03-17T21:46:01.3407642Z Running tests... 2022-03-17T21:46:01.3409353Z ---------------------------------------------------------------------- 2022-03-17T21:46:01.9396868Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85253 2022-03-17T21:46:01.9436698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85254 2022-03-17T21:46:01.9438139Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:03.0732261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:03.0733989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:03.0735093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:03.0736544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:03.0746616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:03.1743810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:05.0523124Z ok (3.711s) 2022-03-17T21:46:05.0523589Z 2022-03-17T21:46:05.0525159Z ---------------------------------------------------------------------- 2022-03-17T21:46:05.0526131Z Ran 1 test in 3.712s 2022-03-17T21:46:05.0526533Z 2022-03-17T21:46:05.0527177Z OK 2022-03-17T21:46:05.0527489Z 2022-03-17T21:46:05.0527804Z Generating XML reports... 2022-03-17T21:46:05.0603867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214601.xml 2022-03-17T21:46:07.6829301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:07.6842743Z 2022-03-17T21:46:07.6843851Z Running tests... 2022-03-17T21:46:07.6845081Z ---------------------------------------------------------------------- 2022-03-17T21:46:08.3078063Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85469 2022-03-17T21:46:08.3174372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85470 2022-03-17T21:46:08.3176437Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:09.4394399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:09.4396811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:09.4398233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:09.4400032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:09.4499492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:09.5415769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:11.6275679Z ok (3.943s) 2022-03-17T21:46:11.6276041Z 2022-03-17T21:46:11.6277001Z ---------------------------------------------------------------------- 2022-03-17T21:46:11.6277798Z Ran 1 test in 3.943s 2022-03-17T21:46:11.6280579Z 2022-03-17T21:46:11.6281064Z OK 2022-03-17T21:46:11.6281484Z 2022-03-17T21:46:11.6281789Z Generating XML reports... 2022-03-17T21:46:11.6349226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214607.xml 2022-03-17T21:46:14.3494604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:14.3504736Z 2022-03-17T21:46:14.3505409Z Running tests... 2022-03-17T21:46:14.3507165Z ---------------------------------------------------------------------- 2022-03-17T21:46:14.9573793Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85685 2022-03-17T21:46:14.9633197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85686 2022-03-17T21:46:14.9634612Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:16.1094692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:16.1096074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:16.1098005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:16.1099787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:16.1107662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:16.2107805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:18.6736360Z ok (4.322s) 2022-03-17T21:46:18.6736933Z 2022-03-17T21:46:18.6738304Z ---------------------------------------------------------------------- 2022-03-17T21:46:18.6739094Z Ran 1 test in 4.323s 2022-03-17T21:46:18.6740117Z 2022-03-17T21:46:18.6740310Z OK 2022-03-17T21:46:18.6743146Z 2022-03-17T21:46:18.6743850Z Generating XML reports... 2022-03-17T21:46:18.6820243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214614.xml 2022-03-17T21:46:21.3060342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:21.3073093Z 2022-03-17T21:46:21.3073335Z Running tests... 2022-03-17T21:46:21.3073802Z ---------------------------------------------------------------------- 2022-03-17T21:46:21.9192053Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85901 2022-03-17T21:46:21.9239672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85902 2022-03-17T21:46:21.9241076Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:23.0739384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:23.0739982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:23.0740996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:23.0741789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:23.0752711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:23.0753174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:24.8341626Z ok (3.526s) 2022-03-17T21:46:24.8342320Z 2022-03-17T21:46:24.8343577Z ---------------------------------------------------------------------- 2022-03-17T21:46:24.8344373Z Ran 1 test in 3.527s 2022-03-17T21:46:24.8344734Z 2022-03-17T21:46:24.8344936Z OK 2022-03-17T21:46:24.8345229Z 2022-03-17T21:46:24.8345503Z Generating XML reports... 2022-03-17T21:46:24.8425439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214621.xml 2022-03-17T21:46:27.5442372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:27.5452019Z 2022-03-17T21:46:27.5452942Z Running tests... 2022-03-17T21:46:27.5453536Z ---------------------------------------------------------------------- 2022-03-17T21:46:28.1449782Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86117 2022-03-17T21:46:28.1484980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86118 2022-03-17T21:46:28.1486373Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:29.3091944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:29.3092571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:29.3093657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:29.3094467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:29.3107093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:29.4104447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:31.5569135Z ok (4.011s) 2022-03-17T21:46:31.5569547Z 2022-03-17T21:46:31.5570766Z ---------------------------------------------------------------------- 2022-03-17T21:46:31.5572545Z Ran 1 test in 4.012s 2022-03-17T21:46:31.5572997Z 2022-03-17T21:46:31.5573238Z OK 2022-03-17T21:46:31.5573604Z 2022-03-17T21:46:31.5573960Z Generating XML reports... 2022-03-17T21:46:31.5651046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214627.xml 2022-03-17T21:46:34.2487202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:34.2499627Z 2022-03-17T21:46:34.2500238Z Running tests... 2022-03-17T21:46:34.2501682Z ---------------------------------------------------------------------- 2022-03-17T21:46:34.8439154Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86327 2022-03-17T21:46:34.8473584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86328 2022-03-17T21:46:34.8475170Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:35.9698741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:35.9699475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:35.9700033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:35.9700799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:35.9813325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:36.0717616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:37.8570654Z ok (3.607s) 2022-03-17T21:46:37.8571086Z 2022-03-17T21:46:37.8572083Z ---------------------------------------------------------------------- 2022-03-17T21:46:37.8573080Z Ran 1 test in 3.607s 2022-03-17T21:46:37.8573538Z 2022-03-17T21:46:37.8575238Z OK 2022-03-17T21:46:37.8575607Z 2022-03-17T21:46:37.8576151Z Generating XML reports... 2022-03-17T21:46:37.8653764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214634.xml 2022-03-17T21:46:40.4999988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:40.5010935Z 2022-03-17T21:46:40.5012584Z Running tests... 2022-03-17T21:46:40.5013840Z ---------------------------------------------------------------------- 2022-03-17T21:46:41.1447538Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86543 2022-03-17T21:46:41.1493289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86544 2022-03-17T21:46:41.1494621Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:42.2890344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:42.2892315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:42.2893667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:42.2895433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:42.2907611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:42.3903582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:44.2566014Z ok (3.755s) 2022-03-17T21:46:44.2566569Z 2022-03-17T21:46:44.2567643Z ---------------------------------------------------------------------- 2022-03-17T21:46:44.2569146Z Ran 1 test in 3.755s 2022-03-17T21:46:44.2569571Z 2022-03-17T21:46:44.2569818Z OK 2022-03-17T21:46:44.2570171Z 2022-03-17T21:46:44.2570448Z Generating XML reports... 2022-03-17T21:46:44.2603671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214640.xml 2022-03-17T21:46:46.9663741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:46.9675765Z 2022-03-17T21:46:46.9676408Z Running tests... 2022-03-17T21:46:46.9677859Z ---------------------------------------------------------------------- 2022-03-17T21:46:47.5765846Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86759 2022-03-17T21:46:47.5812393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86760 2022-03-17T21:46:47.5813914Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:48.7369671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:48.7370737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:48.7371327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:48.7372335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:48.7480224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:48.8393194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:51.2912237Z ok (4.323s) 2022-03-17T21:46:51.2912788Z 2022-03-17T21:46:51.2913954Z ---------------------------------------------------------------------- 2022-03-17T21:46:51.2914915Z Ran 1 test in 4.323s 2022-03-17T21:46:51.2915418Z 2022-03-17T21:46:51.2915607Z OK 2022-03-17T21:46:51.2915906Z 2022-03-17T21:46:51.2916187Z Generating XML reports... 2022-03-17T21:46:51.2988263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214646.xml 2022-03-17T21:46:53.9756018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:46:53.9765216Z 2022-03-17T21:46:53.9765497Z Running tests... 2022-03-17T21:46:53.9765922Z ---------------------------------------------------------------------- 2022-03-17T21:46:54.5761020Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86975 2022-03-17T21:46:54.5799146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86976 2022-03-17T21:46:54.5800487Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:46:55.7298491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:46:55.7300402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:55.7301699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:46:55.7303380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:46:55.7316310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:46:55.7317497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:46:58.1907340Z ok (4.213s) 2022-03-17T21:46:58.1907720Z 2022-03-17T21:46:58.1909407Z ---------------------------------------------------------------------- 2022-03-17T21:46:58.1910182Z Ran 1 test in 4.214s 2022-03-17T21:46:58.1910567Z 2022-03-17T21:46:58.1910754Z OK 2022-03-17T21:46:58.1911043Z 2022-03-17T21:46:58.1911315Z Generating XML reports... 2022-03-17T21:46:58.1986138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214653.xml 2022-03-17T21:47:01.0492091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:01.0504011Z 2022-03-17T21:47:01.0504421Z Running tests... 2022-03-17T21:47:01.0505480Z ---------------------------------------------------------------------- 2022-03-17T21:47:01.6479002Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87191 2022-03-17T21:47:01.6526489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87192 2022-03-17T21:47:01.6528557Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:47:02.7998946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:47:02.8004766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:47:02.8007193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:47:02.8106163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:47:02.8112052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:47:02.9028329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:47:05.1653004Z ok (4.114s) 2022-03-17T21:47:05.1653395Z 2022-03-17T21:47:05.1654278Z ---------------------------------------------------------------------- 2022-03-17T21:47:05.1655044Z Ran 1 test in 4.115s 2022-03-17T21:47:05.1655426Z 2022-03-17T21:47:05.1655633Z OK 2022-03-17T21:47:05.1655934Z 2022-03-17T21:47:05.1656212Z Generating XML reports... 2022-03-17T21:47:05.1724718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214701.xml 2022-03-17T21:47:07.7614781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:07.7624569Z 2022-03-17T21:47:07.7624912Z Running tests... 2022-03-17T21:47:07.7625746Z ---------------------------------------------------------------------- 2022-03-17T21:47:08.3595026Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87407 2022-03-17T21:47:08.3630366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87408 2022-03-17T21:47:08.3631735Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:47:09.5187896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:47:09.5189157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:47:09.5191054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:47:09.5192867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:47:09.5194184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:47:09.5195354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:47:11.8723659Z ok (4.109s) 2022-03-17T21:47:11.8724500Z 2022-03-17T21:47:11.8725383Z ---------------------------------------------------------------------- 2022-03-17T21:47:11.8726146Z Ran 1 test in 4.110s 2022-03-17T21:47:11.8726492Z 2022-03-17T21:47:11.8726683Z OK 2022-03-17T21:47:11.8726969Z 2022-03-17T21:47:11.8727249Z Generating XML reports... 2022-03-17T21:47:11.8799766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214707.xml 2022-03-17T21:47:14.4095609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:14.4106672Z 2022-03-17T21:47:14.4106944Z Running tests... 2022-03-17T21:47:14.4108053Z ---------------------------------------------------------------------- 2022-03-17T21:47:14.4121349Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:14.4122344Z 2022-03-17T21:47:14.4123159Z ---------------------------------------------------------------------- 2022-03-17T21:47:14.4123970Z Ran 1 test in 0.002s 2022-03-17T21:47:14.4124342Z 2022-03-17T21:47:14.4124566Z OK (skipped=1) 2022-03-17T21:47:14.4124920Z 2022-03-17T21:47:14.4125194Z Generating XML reports... 2022-03-17T21:47:14.4163644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214714.xml 2022-03-17T21:47:16.9019463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:16.9028739Z 2022-03-17T21:47:16.9029038Z Running tests... 2022-03-17T21:47:16.9030168Z ---------------------------------------------------------------------- 2022-03-17T21:47:16.9046964Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:16.9047616Z 2022-03-17T21:47:16.9048417Z ---------------------------------------------------------------------- 2022-03-17T21:47:16.9049212Z Ran 1 test in 0.002s 2022-03-17T21:47:16.9049580Z 2022-03-17T21:47:16.9049795Z OK (skipped=1) 2022-03-17T21:47:16.9050170Z 2022-03-17T21:47:16.9050441Z Generating XML reports... 2022-03-17T21:47:16.9085931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214716.xml 2022-03-17T21:47:19.4365412Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:19.4375494Z 2022-03-17T21:47:19.4375987Z Running tests... 2022-03-17T21:47:19.4377175Z ---------------------------------------------------------------------- 2022-03-17T21:47:19.4395067Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:19.4395696Z 2022-03-17T21:47:19.4396505Z ---------------------------------------------------------------------- 2022-03-17T21:47:19.4397255Z Ran 1 test in 0.002s 2022-03-17T21:47:19.4397615Z 2022-03-17T21:47:19.4397844Z OK (skipped=1) 2022-03-17T21:47:19.4398189Z 2022-03-17T21:47:19.4398435Z Generating XML reports... 2022-03-17T21:47:19.4435646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214719.xml 2022-03-17T21:47:21.9888784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:21.9900887Z 2022-03-17T21:47:21.9901393Z Running tests... 2022-03-17T21:47:21.9902369Z ---------------------------------------------------------------------- 2022-03-17T21:47:21.9916834Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:21.9917648Z 2022-03-17T21:47:21.9918325Z ---------------------------------------------------------------------- 2022-03-17T21:47:21.9918965Z Ran 1 test in 0.002s 2022-03-17T21:47:21.9919280Z 2022-03-17T21:47:21.9919475Z OK (skipped=1) 2022-03-17T21:47:21.9919824Z 2022-03-17T21:47:21.9920091Z Generating XML reports... 2022-03-17T21:47:21.9957755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214721.xml 2022-03-17T21:47:24.6064925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:24.6075607Z 2022-03-17T21:47:24.6075917Z Running tests... 2022-03-17T21:47:24.6076346Z ---------------------------------------------------------------------- 2022-03-17T21:47:24.6090288Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:24.6090552Z 2022-03-17T21:47:24.6090851Z ---------------------------------------------------------------------- 2022-03-17T21:47:24.6091141Z Ran 1 test in 0.002s 2022-03-17T21:47:24.6091282Z 2022-03-17T21:47:24.6091371Z OK (skipped=1) 2022-03-17T21:47:24.6091513Z 2022-03-17T21:47:24.6091613Z Generating XML reports... 2022-03-17T21:47:24.6131737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214724.xml 2022-03-17T21:47:27.2069030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:27.2079665Z 2022-03-17T21:47:27.2080386Z Running tests... 2022-03-17T21:47:27.2082303Z ---------------------------------------------------------------------- 2022-03-17T21:47:27.2097690Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:27.2098513Z 2022-03-17T21:47:27.2099478Z ---------------------------------------------------------------------- 2022-03-17T21:47:27.2100450Z Ran 1 test in 0.002s 2022-03-17T21:47:27.2100822Z 2022-03-17T21:47:27.2101063Z OK (skipped=1) 2022-03-17T21:47:27.2101422Z 2022-03-17T21:47:27.2101705Z Generating XML reports... 2022-03-17T21:47:27.2138531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214727.xml 2022-03-17T21:47:29.6764749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:29.6776213Z 2022-03-17T21:47:29.6776581Z Running tests... 2022-03-17T21:47:29.6777115Z ---------------------------------------------------------------------- 2022-03-17T21:47:29.6793069Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:29.6793452Z 2022-03-17T21:47:29.6793819Z ---------------------------------------------------------------------- 2022-03-17T21:47:29.6794176Z Ran 1 test in 0.002s 2022-03-17T21:47:29.6794347Z 2022-03-17T21:47:29.6794457Z OK (skipped=1) 2022-03-17T21:47:29.6794645Z 2022-03-17T21:47:29.6794836Z Generating XML reports... 2022-03-17T21:47:29.6832674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214729.xml 2022-03-17T21:47:32.8080190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:32.8091975Z 2022-03-17T21:47:32.8092423Z Running tests... 2022-03-17T21:47:32.8093618Z ---------------------------------------------------------------------- 2022-03-17T21:47:32.8107526Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:32.8108571Z 2022-03-17T21:47:32.8109597Z ---------------------------------------------------------------------- 2022-03-17T21:47:32.8110571Z Ran 1 test in 0.002s 2022-03-17T21:47:32.8111023Z 2022-03-17T21:47:32.8111315Z OK (skipped=1) 2022-03-17T21:47:32.8111764Z 2022-03-17T21:47:32.8112052Z Generating XML reports... 2022-03-17T21:47:32.8149197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214732.xml 2022-03-17T21:47:35.4405612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:35.4421947Z 2022-03-17T21:47:35.4422690Z Running tests... 2022-03-17T21:47:35.4424164Z ---------------------------------------------------------------------- 2022-03-17T21:47:35.4437851Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:35.4438717Z 2022-03-17T21:47:35.4439707Z ---------------------------------------------------------------------- 2022-03-17T21:47:35.4440995Z Ran 1 test in 0.002s 2022-03-17T21:47:35.4441350Z 2022-03-17T21:47:35.4441588Z OK (skipped=1) 2022-03-17T21:47:35.4442131Z 2022-03-17T21:47:35.4442413Z Generating XML reports... 2022-03-17T21:47:35.4482056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214735.xml 2022-03-17T21:47:38.1733655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:38.1746208Z 2022-03-17T21:47:38.1746454Z Running tests... 2022-03-17T21:47:38.1746892Z ---------------------------------------------------------------------- 2022-03-17T21:47:38.1762680Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:38.1762962Z 2022-03-17T21:47:38.1763345Z ---------------------------------------------------------------------- 2022-03-17T21:47:38.1763735Z Ran 1 test in 0.002s 2022-03-17T21:47:38.1763996Z 2022-03-17T21:47:38.1764102Z OK (skipped=1) 2022-03-17T21:47:38.1765401Z 2022-03-17T21:47:38.1765837Z Generating XML reports... 2022-03-17T21:47:38.1801171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214738.xml 2022-03-17T21:47:40.6812189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:40.6826836Z 2022-03-17T21:47:40.6827640Z Running tests... 2022-03-17T21:47:40.6829053Z ---------------------------------------------------------------------- 2022-03-17T21:47:40.6846029Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:40.6846947Z 2022-03-17T21:47:40.6847913Z ---------------------------------------------------------------------- 2022-03-17T21:47:40.6848871Z Ran 1 test in 0.002s 2022-03-17T21:47:40.6849326Z 2022-03-17T21:47:40.6849563Z OK (skipped=1) 2022-03-17T21:47:40.6849914Z 2022-03-17T21:47:40.6850212Z Generating XML reports... 2022-03-17T21:47:40.6890280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214740.xml 2022-03-17T21:47:43.4105170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:43.4109495Z 2022-03-17T21:47:43.4110617Z Running tests... 2022-03-17T21:47:43.4112355Z ---------------------------------------------------------------------- 2022-03-17T21:47:43.4129824Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:43.4130735Z 2022-03-17T21:47:43.4131627Z ---------------------------------------------------------------------- 2022-03-17T21:47:43.4132442Z Ran 1 test in 0.002s 2022-03-17T21:47:43.4132861Z 2022-03-17T21:47:43.4133132Z OK (skipped=1) 2022-03-17T21:47:43.4133530Z 2022-03-17T21:47:43.4133837Z Generating XML reports... 2022-03-17T21:47:43.4166474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214743.xml 2022-03-17T21:47:45.8786571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:45.8795521Z 2022-03-17T21:47:45.8796077Z Running tests... 2022-03-17T21:47:45.8797234Z ---------------------------------------------------------------------- 2022-03-17T21:47:45.8812048Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:45.8813057Z 2022-03-17T21:47:45.8813960Z ---------------------------------------------------------------------- 2022-03-17T21:47:45.8814768Z Ran 1 test in 0.002s 2022-03-17T21:47:45.8815180Z 2022-03-17T21:47:45.8815444Z OK (skipped=1) 2022-03-17T21:47:45.8815826Z 2022-03-17T21:47:45.8816090Z Generating XML reports... 2022-03-17T21:47:45.8851976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214745.xml 2022-03-17T21:47:48.3927688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:48.3939234Z 2022-03-17T21:47:48.3939851Z Running tests... 2022-03-17T21:47:48.3941073Z ---------------------------------------------------------------------- 2022-03-17T21:47:48.3957181Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:48.3958012Z 2022-03-17T21:47:48.3958871Z ---------------------------------------------------------------------- 2022-03-17T21:47:48.3959673Z Ran 1 test in 0.002s 2022-03-17T21:47:48.3960071Z 2022-03-17T21:47:48.3960338Z OK (skipped=1) 2022-03-17T21:47:48.3960722Z 2022-03-17T21:47:48.3960977Z Generating XML reports... 2022-03-17T21:47:48.3997796Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214748.xml 2022-03-17T21:47:51.4030230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:51.4038422Z 2022-03-17T21:47:51.4038846Z Running tests... 2022-03-17T21:47:51.4039830Z ---------------------------------------------------------------------- 2022-03-17T21:47:51.4057705Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T21:47:51.4058441Z 2022-03-17T21:47:51.4059282Z ---------------------------------------------------------------------- 2022-03-17T21:47:51.4060066Z Ran 1 test in 0.002s 2022-03-17T21:47:51.4060457Z 2022-03-17T21:47:51.4060728Z OK (skipped=1) 2022-03-17T21:47:51.4061103Z 2022-03-17T21:47:51.4061354Z Generating XML reports... 2022-03-17T21:47:51.4096632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214751.xml 2022-03-17T21:47:53.9471165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:53.9479019Z 2022-03-17T21:47:53.9479301Z Running tests... 2022-03-17T21:47:53.9481667Z ---------------------------------------------------------------------- 2022-03-17T21:47:53.9500480Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:53.9501233Z 2022-03-17T21:47:53.9501995Z ---------------------------------------------------------------------- 2022-03-17T21:47:53.9502898Z Ran 1 test in 0.002s 2022-03-17T21:47:53.9503285Z 2022-03-17T21:47:53.9503544Z OK (skipped=1) 2022-03-17T21:47:53.9503876Z 2022-03-17T21:47:53.9504536Z Generating XML reports... 2022-03-17T21:47:53.9543918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214753.xml 2022-03-17T21:47:56.6503486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:56.6512912Z 2022-03-17T21:47:56.6513491Z Running tests... 2022-03-17T21:47:56.6516064Z ---------------------------------------------------------------------- 2022-03-17T21:47:56.6536100Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:56.6537008Z 2022-03-17T21:47:56.6539412Z ---------------------------------------------------------------------- 2022-03-17T21:47:56.6540933Z Ran 1 test in 0.002s 2022-03-17T21:47:56.6541465Z 2022-03-17T21:47:56.6541726Z OK (skipped=1) 2022-03-17T21:47:56.6542058Z 2022-03-17T21:47:56.6542363Z Generating XML reports... 2022-03-17T21:47:56.6572706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214756.xml 2022-03-17T21:47:59.1249288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:47:59.1260531Z 2022-03-17T21:47:59.1260918Z Running tests... 2022-03-17T21:47:59.1261581Z ---------------------------------------------------------------------- 2022-03-17T21:47:59.1279753Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:47:59.1280297Z 2022-03-17T21:47:59.1280887Z ---------------------------------------------------------------------- 2022-03-17T21:47:59.1281296Z Ran 1 test in 0.002s 2022-03-17T21:47:59.1281494Z 2022-03-17T21:47:59.1281589Z OK (skipped=1) 2022-03-17T21:47:59.1281768Z 2022-03-17T21:47:59.1282033Z Generating XML reports... 2022-03-17T21:47:59.1320669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214759.xml 2022-03-17T21:48:01.6005616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:01.6018559Z 2022-03-17T21:48:01.6018836Z Running tests... 2022-03-17T21:48:01.6021218Z ---------------------------------------------------------------------- 2022-03-17T21:48:01.6038773Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:01.6039133Z 2022-03-17T21:48:01.6039519Z ---------------------------------------------------------------------- 2022-03-17T21:48:01.6039893Z Ran 1 test in 0.002s 2022-03-17T21:48:01.6040077Z 2022-03-17T21:48:01.6040213Z OK (skipped=1) 2022-03-17T21:48:01.6040389Z 2022-03-17T21:48:01.6040546Z Generating XML reports... 2022-03-17T21:48:01.6078142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214801.xml 2022-03-17T21:48:04.1201003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:04.1215033Z 2022-03-17T21:48:04.1215383Z Running tests... 2022-03-17T21:48:04.1215870Z ---------------------------------------------------------------------- 2022-03-17T21:48:04.1234733Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:04.1235055Z 2022-03-17T21:48:04.1235798Z ---------------------------------------------------------------------- 2022-03-17T21:48:04.1236280Z Ran 1 test in 0.002s 2022-03-17T21:48:04.1236495Z 2022-03-17T21:48:04.1236618Z OK (skipped=1) 2022-03-17T21:48:04.1236804Z 2022-03-17T21:48:04.1236926Z Generating XML reports... 2022-03-17T21:48:04.1275712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214804.xml 2022-03-17T21:48:06.6514355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:06.6528114Z 2022-03-17T21:48:06.6528515Z Running tests... 2022-03-17T21:48:06.6529810Z ---------------------------------------------------------------------- 2022-03-17T21:48:06.6547640Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:06.6548537Z 2022-03-17T21:48:06.6549614Z ---------------------------------------------------------------------- 2022-03-17T21:48:06.6550525Z Ran 1 test in 0.002s 2022-03-17T21:48:06.6550959Z 2022-03-17T21:48:06.6551194Z OK (skipped=1) 2022-03-17T21:48:06.6551605Z 2022-03-17T21:48:06.6551945Z Generating XML reports... 2022-03-17T21:48:06.6586423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214806.xml 2022-03-17T21:48:09.1235290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:09.1250917Z 2022-03-17T21:48:09.1251241Z Running tests... 2022-03-17T21:48:09.1251809Z ---------------------------------------------------------------------- 2022-03-17T21:48:09.1271464Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:09.1271927Z 2022-03-17T21:48:09.1272415Z ---------------------------------------------------------------------- 2022-03-17T21:48:09.1272890Z Ran 1 test in 0.002s 2022-03-17T21:48:09.1273092Z 2022-03-17T21:48:09.1273254Z OK (skipped=1) 2022-03-17T21:48:09.1273460Z 2022-03-17T21:48:09.1273610Z Generating XML reports... 2022-03-17T21:48:09.1309235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214809.xml 2022-03-17T21:48:11.5945735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:11.5959062Z 2022-03-17T21:48:11.5959534Z Running tests... 2022-03-17T21:48:11.5960024Z ---------------------------------------------------------------------- 2022-03-17T21:48:11.5978151Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:11.5979059Z 2022-03-17T21:48:11.5980096Z ---------------------------------------------------------------------- 2022-03-17T21:48:11.5981211Z Ran 1 test in 0.002s 2022-03-17T21:48:11.5981709Z 2022-03-17T21:48:11.5981986Z OK (skipped=1) 2022-03-17T21:48:11.5982354Z 2022-03-17T21:48:11.5982675Z Generating XML reports... 2022-03-17T21:48:11.6018723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214811.xml 2022-03-17T21:48:14.3376677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:14.3390180Z 2022-03-17T21:48:14.3390424Z Running tests... 2022-03-17T21:48:14.3409425Z ---------------------------------------------------------------------- 2022-03-17T21:48:14.3410837Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:14.3411725Z 2022-03-17T21:48:14.3412583Z ---------------------------------------------------------------------- 2022-03-17T21:48:14.3413383Z Ran 1 test in 0.002s 2022-03-17T21:48:14.3413777Z 2022-03-17T21:48:14.3414038Z OK (skipped=1) 2022-03-17T21:48:14.3414422Z 2022-03-17T21:48:14.3414684Z Generating XML reports... 2022-03-17T21:48:14.3452993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214814.xml 2022-03-17T21:48:16.8722152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:16.8736213Z 2022-03-17T21:48:16.8736991Z Running tests... 2022-03-17T21:48:16.8738237Z ---------------------------------------------------------------------- 2022-03-17T21:48:17.4746406Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89231 2022-03-17T21:48:17.4773619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89232 2022-03-17T21:48:17.4775667Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:48:18.6333324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:48:18.6334849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:48:18.6337010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:18.6338942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:18.6441300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:48:18.7346486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:48:19.5504251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:48:19.5638026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:48:19.5639451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:48:19.5706921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:48:19.6073891Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:48:19.6611978Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T21:48:21.3879453Z ok (4.514s) 2022-03-17T21:48:21.3880005Z 2022-03-17T21:48:21.3881242Z ---------------------------------------------------------------------- 2022-03-17T21:48:21.3882422Z Ran 1 test in 4.514s 2022-03-17T21:48:21.3882932Z 2022-03-17T21:48:21.3883211Z OK 2022-03-17T21:48:21.3883676Z 2022-03-17T21:48:21.3884134Z Generating XML reports... 2022-03-17T21:48:21.3947569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214816.xml 2022-03-17T21:48:23.9630945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:23.9644418Z 2022-03-17T21:48:23.9644889Z Running tests... 2022-03-17T21:48:23.9645431Z ---------------------------------------------------------------------- 2022-03-17T21:48:24.5629729Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89452 2022-03-17T21:48:24.5663836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89453 2022-03-17T21:48:24.5665166Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:48:25.7015093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:48:25.7016648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:48:25.7018821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:25.7020645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:25.7124636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:48:25.8027242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:48:27.4758889Z skip (3.511s) 2022-03-17T21:48:27.4759500Z 2022-03-17T21:48:27.4760637Z ---------------------------------------------------------------------- 2022-03-17T21:48:27.4762258Z Ran 1 test in 3.511s 2022-03-17T21:48:27.4762929Z 2022-03-17T21:48:27.4763282Z OK (skipped=1) 2022-03-17T21:48:27.4763786Z 2022-03-17T21:48:27.4764529Z Generating XML reports... 2022-03-17T21:48:27.4841744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214823.xml 2022-03-17T21:48:30.0971208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:30.0982330Z 2022-03-17T21:48:30.0982467Z Running tests... 2022-03-17T21:48:30.0983537Z ---------------------------------------------------------------------- 2022-03-17T21:48:30.0998780Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:48:30.0999033Z 2022-03-17T21:48:30.0999377Z ---------------------------------------------------------------------- 2022-03-17T21:48:30.0999673Z Ran 1 test in 0.002s 2022-03-17T21:48:30.0999830Z 2022-03-17T21:48:30.0999933Z OK (skipped=1) 2022-03-17T21:48:30.1000076Z 2022-03-17T21:48:30.1000169Z Generating XML reports... 2022-03-17T21:48:30.1040497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214830.xml 2022-03-17T21:48:32.6874631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:32.6886383Z 2022-03-17T21:48:32.6886674Z Running tests... 2022-03-17T21:48:32.6887181Z ---------------------------------------------------------------------- 2022-03-17T21:48:33.2950014Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89729 2022-03-17T21:48:33.3012919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89730 2022-03-17T21:48:33.3014368Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:48:34.4700999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:48:34.4703168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:34.4704604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:48:34.4706388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:34.4716515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:48:34.5717186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:48:37.1115190Z ok (4.422s) 2022-03-17T21:48:37.1115678Z 2022-03-17T21:48:37.1116601Z ---------------------------------------------------------------------- 2022-03-17T21:48:37.1117432Z Ran 1 test in 4.423s 2022-03-17T21:48:37.1117779Z 2022-03-17T21:48:37.1118005Z OK 2022-03-17T21:48:37.1118322Z 2022-03-17T21:48:37.1118620Z Generating XML reports... 2022-03-17T21:48:37.1188728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214832.xml 2022-03-17T21:48:40.0662183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:40.0672543Z 2022-03-17T21:48:40.0673095Z Running tests... 2022-03-17T21:48:40.6678423Z ---------------------------------------------------------------------- 2022-03-17T21:48:40.6679888Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89939 2022-03-17T21:48:40.6716467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89940 2022-03-17T21:48:40.6717812Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:48:41.8151946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:48:41.8153877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:48:41.8156148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:41.8158169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:41.8159472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:48:41.8160704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:48:44.7816014Z ok (4.714s) 2022-03-17T21:48:44.7816535Z 2022-03-17T21:48:44.7817865Z ---------------------------------------------------------------------- 2022-03-17T21:48:44.7818755Z Ran 1 test in 4.714s 2022-03-17T21:48:44.7819147Z 2022-03-17T21:48:44.7819324Z OK 2022-03-17T21:48:44.7819647Z 2022-03-17T21:48:44.7819964Z Generating XML reports... 2022-03-17T21:48:44.7899045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214840.xml 2022-03-17T21:48:47.4555336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:47.4567659Z 2022-03-17T21:48:47.4568171Z Running tests... 2022-03-17T21:48:47.4569481Z ---------------------------------------------------------------------- 2022-03-17T21:48:48.0546741Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90149 2022-03-17T21:48:48.0579156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90150 2022-03-17T21:48:48.0580541Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:48:49.1974080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:48:49.1976054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:49.1977490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:48:49.1979288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:49.2089075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:48:49.2994095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:48:49.3204085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:48:49.3205411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:48:49.3207353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:48:49.3209305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:48:51.8719770Z ok (4.415s) 2022-03-17T21:48:51.8720192Z 2022-03-17T21:48:51.8721107Z ---------------------------------------------------------------------- 2022-03-17T21:48:51.8722122Z Ran 1 test in 4.415s 2022-03-17T21:48:51.8722534Z 2022-03-17T21:48:51.8722788Z OK 2022-03-17T21:48:51.8723106Z 2022-03-17T21:48:51.8723392Z Generating XML reports... 2022-03-17T21:48:51.8801294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214847.xml 2022-03-17T21:48:54.5170210Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:48:54.5181992Z 2022-03-17T21:48:54.5182886Z Running tests... 2022-03-17T21:48:54.5184172Z ---------------------------------------------------------------------- 2022-03-17T21:48:55.1155284Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90365 2022-03-17T21:48:55.1183482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90366 2022-03-17T21:48:55.1185134Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:48:56.2686277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:48:56.2687898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:48:56.2689987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:56.2691904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:48:56.2793669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:48:56.3698718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:48:58.0267570Z skip (3.508s) 2022-03-17T21:48:58.0268157Z 2022-03-17T21:48:58.0269203Z ---------------------------------------------------------------------- 2022-03-17T21:48:58.0270270Z Ran 1 test in 3.509s 2022-03-17T21:48:58.0270958Z 2022-03-17T21:48:58.0271280Z OK (skipped=1) 2022-03-17T21:48:58.0271612Z 2022-03-17T21:48:58.0272316Z Generating XML reports... 2022-03-17T21:48:58.0337357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214854.xml 2022-03-17T21:49:00.6650655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:00.6662267Z 2022-03-17T21:49:00.6662882Z Running tests... 2022-03-17T21:49:00.6663969Z ---------------------------------------------------------------------- 2022-03-17T21:49:01.2898044Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90575 2022-03-17T21:49:01.2957672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90576 2022-03-17T21:49:01.2959185Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:02.4536111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:02.4538473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:02.4539903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:02.4541672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:02.4644701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:02.5563656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:04.3022402Z skip (3.635s) 2022-03-17T21:49:04.3022917Z 2022-03-17T21:49:04.3024124Z ---------------------------------------------------------------------- 2022-03-17T21:49:04.3025187Z Ran 1 test in 3.636s 2022-03-17T21:49:04.3025621Z 2022-03-17T21:49:04.3025995Z OK (skipped=1) 2022-03-17T21:49:04.3026436Z 2022-03-17T21:49:04.3026743Z Generating XML reports... 2022-03-17T21:49:04.3085712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214900.xml 2022-03-17T21:49:06.9385493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:06.9398001Z 2022-03-17T21:49:06.9398706Z Running tests... 2022-03-17T21:49:06.9400357Z ---------------------------------------------------------------------- 2022-03-17T21:49:07.5397979Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90785 2022-03-17T21:49:07.5436736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90786 2022-03-17T21:49:07.5438359Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:08.6945496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:08.6947030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:08.6949301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:08.7049502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:08.7052215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:08.7960317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:10.5520007Z skip (3.612s) 2022-03-17T21:49:10.5520724Z 2022-03-17T21:49:10.5521806Z ---------------------------------------------------------------------- 2022-03-17T21:49:10.5523100Z Ran 1 test in 3.612s 2022-03-17T21:49:10.5523594Z 2022-03-17T21:49:10.5523892Z OK (skipped=1) 2022-03-17T21:49:10.5524711Z 2022-03-17T21:49:10.5525038Z Generating XML reports... 2022-03-17T21:49:10.5604603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214906.xml 2022-03-17T21:49:13.1399615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:13.1411636Z 2022-03-17T21:49:13.1411917Z Running tests... 2022-03-17T21:49:13.1412360Z ---------------------------------------------------------------------- 2022-03-17T21:49:13.7406732Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90995 2022-03-17T21:49:13.7439080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90996 2022-03-17T21:49:13.7440412Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:14.8957944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:14.8958482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:14.8959224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:14.9063541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:14.9066935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:15.0070687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:15.0184078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:49:15.0184556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:49:15.0185262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:49:15.0287816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:49:17.8556207Z ok (4.714s) 2022-03-17T21:49:17.8591427Z 2022-03-17T21:49:17.8592531Z ---------------------------------------------------------------------- 2022-03-17T21:49:17.8593958Z Ran 1 test in 4.714s 2022-03-17T21:49:17.8594386Z 2022-03-17T21:49:17.8594603Z OK 2022-03-17T21:49:17.8594956Z 2022-03-17T21:49:17.8595265Z Generating XML reports... 2022-03-17T21:49:17.8637519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214913.xml 2022-03-17T21:49:20.4111377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:20.4123321Z 2022-03-17T21:49:20.4123951Z Running tests... 2022-03-17T21:49:20.4124992Z ---------------------------------------------------------------------- 2022-03-17T21:49:21.0133154Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91211 2022-03-17T21:49:21.0182542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91212 2022-03-17T21:49:21.0183932Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:22.1604643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:22.1605247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:22.1606147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:22.1606976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:22.1723353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:22.2625762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:22.3031773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:22.3032367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:22.3033238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:22.3034054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:25.5292229Z ok (5.116s) 2022-03-17T21:49:25.5292809Z 2022-03-17T21:49:25.5294162Z ---------------------------------------------------------------------- 2022-03-17T21:49:25.5295099Z Ran 1 test in 5.117s 2022-03-17T21:49:25.5295450Z 2022-03-17T21:49:25.5295663Z OK 2022-03-17T21:49:25.5295982Z 2022-03-17T21:49:25.5296279Z Generating XML reports... 2022-03-17T21:49:25.5370320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214920.xml 2022-03-17T21:49:28.1802674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:28.1814326Z 2022-03-17T21:49:28.1814615Z Running tests... 2022-03-17T21:49:28.1815104Z ---------------------------------------------------------------------- 2022-03-17T21:49:28.7753685Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91427 2022-03-17T21:49:28.7787385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91428 2022-03-17T21:49:28.7788765Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:29.9158225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:29.9160315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:29.9162267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:29.9164062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:29.9266216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:30.0177975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:31.7867605Z skip (3.605s) 2022-03-17T21:49:31.7867986Z 2022-03-17T21:49:31.7869303Z ---------------------------------------------------------------------- 2022-03-17T21:49:31.7870172Z Ran 1 test in 3.605s 2022-03-17T21:49:31.7870565Z 2022-03-17T21:49:31.7870796Z OK (skipped=1) 2022-03-17T21:49:31.7871167Z 2022-03-17T21:49:31.7871468Z Generating XML reports... 2022-03-17T21:49:31.7949437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214928.xml 2022-03-17T21:49:34.4373141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:34.4386407Z 2022-03-17T21:49:34.4388168Z Running tests... 2022-03-17T21:49:34.4389770Z ---------------------------------------------------------------------- 2022-03-17T21:49:35.0693252Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91637 2022-03-17T21:49:35.0748951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91638 2022-03-17T21:49:35.0750785Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:36.2124412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:36.2125816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:36.2127774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:36.2129561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:36.2130917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:36.2132875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:38.0834907Z ok (3.644s) 2022-03-17T21:49:38.0835313Z 2022-03-17T21:49:38.0836433Z ---------------------------------------------------------------------- 2022-03-17T21:49:38.0837460Z Ran 1 test in 3.645s 2022-03-17T21:49:38.0838122Z 2022-03-17T21:49:38.0838340Z OK 2022-03-17T21:49:38.0838788Z 2022-03-17T21:49:38.0843105Z Generating XML reports... 2022-03-17T21:49:38.0915675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214934.xml 2022-03-17T21:49:40.7855000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:40.7868319Z 2022-03-17T21:49:40.7868886Z Running tests... 2022-03-17T21:49:40.7869938Z ---------------------------------------------------------------------- 2022-03-17T21:49:41.3840267Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91847 2022-03-17T21:49:41.3880130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91848 2022-03-17T21:49:41.3881537Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:49:42.4987142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:49:42.4988442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:49:42.4990890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:42.5090995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:49:42.5092504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:49:42.6013285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:49:44.3977564Z ok (3.611s) 2022-03-17T21:49:44.3978356Z 2022-03-17T21:49:44.3979671Z ---------------------------------------------------------------------- 2022-03-17T21:49:44.3980568Z Ran 1 test in 3.611s 2022-03-17T21:49:44.3980964Z 2022-03-17T21:49:44.3981133Z OK 2022-03-17T21:49:44.3981466Z 2022-03-17T21:49:44.3982131Z Generating XML reports... 2022-03-17T21:49:44.4060994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214940.xml 2022-03-17T21:49:46.9938117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:46.9950609Z 2022-03-17T21:49:46.9950875Z Running tests... 2022-03-17T21:49:46.9974262Z ---------------------------------------------------------------------- 2022-03-17T21:49:46.9974868Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:49:46.9975180Z 2022-03-17T21:49:46.9975529Z ---------------------------------------------------------------------- 2022-03-17T21:49:46.9976095Z Ran 1 test in 0.002s 2022-03-17T21:49:46.9976244Z 2022-03-17T21:49:46.9976372Z OK (skipped=1) 2022-03-17T21:49:46.9976544Z 2022-03-17T21:49:46.9976655Z Generating XML reports... 2022-03-17T21:49:47.0016148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214946.xml 2022-03-17T21:49:49.5204958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:49.5217820Z 2022-03-17T21:49:49.5218408Z Running tests... 2022-03-17T21:49:49.5218859Z ---------------------------------------------------------------------- 2022-03-17T21:49:49.5242088Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:49:49.5242560Z 2022-03-17T21:49:49.5243201Z ---------------------------------------------------------------------- 2022-03-17T21:49:49.5243639Z Ran 1 test in 0.003s 2022-03-17T21:49:49.5243826Z 2022-03-17T21:49:49.5243934Z OK (skipped=1) 2022-03-17T21:49:49.5244129Z 2022-03-17T21:49:49.5244288Z Generating XML reports... 2022-03-17T21:49:49.5282641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214949.xml 2022-03-17T21:49:52.1526290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:52.1543174Z 2022-03-17T21:49:52.1543481Z Running tests... 2022-03-17T21:49:52.1544160Z ---------------------------------------------------------------------- 2022-03-17T21:49:52.1574337Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:49:52.1575059Z 2022-03-17T21:49:52.1576082Z ---------------------------------------------------------------------- 2022-03-17T21:49:52.1577081Z Ran 1 test in 0.003s 2022-03-17T21:49:52.1577451Z 2022-03-17T21:49:52.1577677Z OK (skipped=1) 2022-03-17T21:49:52.1578055Z 2022-03-17T21:49:52.1578361Z Generating XML reports... 2022-03-17T21:49:52.1616161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214952.xml 2022-03-17T21:49:54.8543430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:54.8558210Z 2022-03-17T21:49:54.8558485Z Running tests... 2022-03-17T21:49:54.8559119Z ---------------------------------------------------------------------- 2022-03-17T21:49:54.8578507Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:49:54.8578857Z 2022-03-17T21:49:54.8579255Z ---------------------------------------------------------------------- 2022-03-17T21:49:54.8579715Z Ran 1 test in 0.002s 2022-03-17T21:49:54.8579895Z 2022-03-17T21:49:54.8580023Z OK (skipped=1) 2022-03-17T21:49:54.8580184Z 2022-03-17T21:49:54.8580326Z Generating XML reports... 2022-03-17T21:49:54.8616686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214954.xml 2022-03-17T21:49:57.3264294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:49:57.3277866Z 2022-03-17T21:49:57.3278538Z Running tests... 2022-03-17T21:49:57.3279723Z ---------------------------------------------------------------------- 2022-03-17T21:49:57.3292999Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:49:57.3293736Z 2022-03-17T21:49:57.3294540Z ---------------------------------------------------------------------- 2022-03-17T21:49:57.3295393Z Ran 1 test in 0.002s 2022-03-17T21:49:57.3295783Z 2022-03-17T21:49:57.3296054Z OK (skipped=1) 2022-03-17T21:49:57.3296388Z 2022-03-17T21:49:57.3296704Z Generating XML reports... 2022-03-17T21:49:57.3333317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214957.xml 2022-03-17T21:50:00.0150246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:00.0165879Z 2022-03-17T21:50:00.0166539Z Running tests... 2022-03-17T21:50:00.0167757Z ---------------------------------------------------------------------- 2022-03-17T21:50:00.0190948Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:50:00.0191846Z 2022-03-17T21:50:00.0193065Z ---------------------------------------------------------------------- 2022-03-17T21:50:00.0193913Z Ran 1 test in 0.003s 2022-03-17T21:50:00.0194333Z 2022-03-17T21:50:00.0194555Z OK (skipped=1) 2022-03-17T21:50:00.0194943Z 2022-03-17T21:50:00.0195247Z Generating XML reports... 2022-03-17T21:50:00.0235822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215000.xml 2022-03-17T21:50:02.7399413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:02.7411245Z 2022-03-17T21:50:02.7411894Z Running tests... 2022-03-17T21:50:02.7413147Z ---------------------------------------------------------------------- 2022-03-17T21:50:02.7429568Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:50:02.7430328Z 2022-03-17T21:50:02.7431166Z ---------------------------------------------------------------------- 2022-03-17T21:50:02.7432023Z Ran 1 test in 0.002s 2022-03-17T21:50:02.7432380Z 2022-03-17T21:50:02.7432655Z OK (skipped=1) 2022-03-17T21:50:02.7433036Z 2022-03-17T21:50:02.7433353Z Generating XML reports... 2022-03-17T21:50:02.7468824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215002.xml 2022-03-17T21:50:05.2172169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:05.2184666Z 2022-03-17T21:50:05.2184941Z Running tests... 2022-03-17T21:50:05.2185381Z ---------------------------------------------------------------------- 2022-03-17T21:50:05.8156932Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92526 2022-03-17T21:50:05.8187144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92527 2022-03-17T21:50:05.8188530Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:06.9108513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:06.9109800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:06.9111714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:06.9113566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:06.9118594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:06.9119634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:08.7275116Z ok (3.508s) 2022-03-17T21:50:08.7275481Z 2022-03-17T21:50:08.7276664Z ---------------------------------------------------------------------- 2022-03-17T21:50:08.7277495Z Ran 1 test in 3.509s 2022-03-17T21:50:08.7277902Z 2022-03-17T21:50:08.7278088Z OK 2022-03-17T21:50:08.7278403Z 2022-03-17T21:50:08.7278734Z Generating XML reports... 2022-03-17T21:50:08.7360072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215005.xml 2022-03-17T21:50:11.2801968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:11.2814295Z 2022-03-17T21:50:11.2814919Z Running tests... 2022-03-17T21:50:11.2815535Z ---------------------------------------------------------------------- 2022-03-17T21:50:11.8804578Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92742 2022-03-17T21:50:11.8842945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92743 2022-03-17T21:50:11.8844278Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:13.0276015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:13.0277267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:13.0279594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:13.0378824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:13.0380237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:13.1303117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:15.9959114Z ok (4.714s) 2022-03-17T21:50:15.9959765Z 2022-03-17T21:50:15.9960986Z ---------------------------------------------------------------------- 2022-03-17T21:50:15.9962172Z Ran 1 test in 4.714s 2022-03-17T21:50:15.9962567Z 2022-03-17T21:50:15.9962809Z OK 2022-03-17T21:50:15.9963203Z 2022-03-17T21:50:15.9963554Z Generating XML reports... 2022-03-17T21:50:16.0038675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215011.xml 2022-03-17T21:50:18.6330196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:18.6343957Z 2022-03-17T21:50:18.6344627Z Running tests... 2022-03-17T21:50:18.6345849Z ---------------------------------------------------------------------- 2022-03-17T21:50:19.2528698Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92958 2022-03-17T21:50:19.2565875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92959 2022-03-17T21:50:19.2567363Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:20.3721407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:20.3723034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:20.3724940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:20.3824085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:20.3844166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:20.4835471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:20.5038363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:50:20.5039920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:50:20.5042156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:50:20.5141273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:50:22.3654468Z ok (3.731s) 2022-03-17T21:50:22.3654937Z 2022-03-17T21:50:22.3656089Z ---------------------------------------------------------------------- 2022-03-17T21:50:22.3656955Z Ran 1 test in 3.731s 2022-03-17T21:50:22.3658129Z 2022-03-17T21:50:22.3658340Z OK 2022-03-17T21:50:22.3658616Z 2022-03-17T21:50:22.3661271Z Generating XML reports... 2022-03-17T21:50:22.3736036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215018.xml 2022-03-17T21:50:25.0070520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:25.0082358Z 2022-03-17T21:50:25.0082961Z Running tests... 2022-03-17T21:50:25.0084160Z ---------------------------------------------------------------------- 2022-03-17T21:50:25.6082648Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93180 2022-03-17T21:50:25.6123221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93181 2022-03-17T21:50:25.6124610Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:26.7930385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:26.7931753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:26.7933678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:26.7935626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:26.7950522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:26.8951384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:28.6204702Z skip (3.612s) 2022-03-17T21:50:28.6205490Z 2022-03-17T21:50:28.6206797Z ---------------------------------------------------------------------- 2022-03-17T21:50:28.6207684Z Ran 1 test in 3.612s 2022-03-17T21:50:28.6208108Z 2022-03-17T21:50:28.6208375Z OK (skipped=1) 2022-03-17T21:50:28.6208757Z 2022-03-17T21:50:28.6209032Z Generating XML reports... 2022-03-17T21:50:28.6288831Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215025.xml 2022-03-17T21:50:31.3269583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:31.3281322Z 2022-03-17T21:50:31.3282158Z Running tests... 2022-03-17T21:50:31.3283810Z ---------------------------------------------------------------------- 2022-03-17T21:50:31.9628059Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93390 2022-03-17T21:50:31.9662743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93391 2022-03-17T21:50:31.9664180Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:33.0973502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:33.0974539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:33.0975296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:33.1076683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:33.1079656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:33.1990496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:36.0770664Z ok (4.748s) 2022-03-17T21:50:36.0771036Z 2022-03-17T21:50:36.0772207Z ---------------------------------------------------------------------- 2022-03-17T21:50:36.0773770Z Ran 1 test in 4.749s 2022-03-17T21:50:36.0774265Z 2022-03-17T21:50:36.0774434Z OK 2022-03-17T21:50:36.0774748Z 2022-03-17T21:50:36.0775053Z Generating XML reports... 2022-03-17T21:50:36.0854948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215031.xml 2022-03-17T21:50:38.7280770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:38.7292538Z 2022-03-17T21:50:38.7293230Z Running tests... 2022-03-17T21:50:38.7294537Z ---------------------------------------------------------------------- 2022-03-17T21:50:39.3276793Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93600 2022-03-17T21:50:39.3316464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93601 2022-03-17T21:50:39.3317871Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:40.4231019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:40.4232382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:40.4234316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:40.4333961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:40.4352869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:40.5346859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:42.3415430Z ok (3.612s) 2022-03-17T21:50:42.3415860Z 2022-03-17T21:50:42.3416883Z ---------------------------------------------------------------------- 2022-03-17T21:50:42.3417739Z Ran 1 test in 3.612s 2022-03-17T21:50:42.3418156Z 2022-03-17T21:50:42.3418396Z OK 2022-03-17T21:50:42.3418711Z 2022-03-17T21:50:42.3419003Z Generating XML reports... 2022-03-17T21:50:42.3497257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215038.xml 2022-03-17T21:50:45.0167880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:45.0179932Z 2022-03-17T21:50:45.0180605Z Running tests... 2022-03-17T21:50:45.0181067Z ---------------------------------------------------------------------- 2022-03-17T21:50:45.6194365Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93810 2022-03-17T21:50:45.6240755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93811 2022-03-17T21:50:45.6242301Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:46.7627312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:46.7645263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:46.7646926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:46.7735337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:46.7749196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:46.8665101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:48.4337686Z skip (3.415s) 2022-03-17T21:50:48.4338061Z 2022-03-17T21:50:48.4339322Z ---------------------------------------------------------------------- 2022-03-17T21:50:48.4340163Z Ran 1 test in 3.415s 2022-03-17T21:50:48.4340568Z 2022-03-17T21:50:48.4340779Z OK (skipped=1) 2022-03-17T21:50:48.4341151Z 2022-03-17T21:50:48.4341441Z Generating XML reports... 2022-03-17T21:50:48.4421598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215045.xml 2022-03-17T21:50:51.0794740Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:51.0804358Z 2022-03-17T21:50:51.0804796Z Running tests... 2022-03-17T21:50:51.0805995Z ---------------------------------------------------------------------- 2022-03-17T21:50:51.6996165Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94020 2022-03-17T21:50:51.7038268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94021 2022-03-17T21:50:51.7039664Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:52.8536688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:52.8537321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:52.8538202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:52.8641410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:52.8646198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:52.9649888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:50:54.7135273Z skip (3.632s) 2022-03-17T21:50:54.7135701Z 2022-03-17T21:50:54.7136564Z ---------------------------------------------------------------------- 2022-03-17T21:50:54.7137357Z Ran 1 test in 3.633s 2022-03-17T21:50:54.7137763Z 2022-03-17T21:50:54.7138021Z OK (skipped=1) 2022-03-17T21:50:54.7138381Z 2022-03-17T21:50:54.7138672Z Generating XML reports... 2022-03-17T21:50:54.7218272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215051.xml 2022-03-17T21:50:57.3702799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:50:57.3712944Z 2022-03-17T21:50:57.3713293Z Running tests... 2022-03-17T21:50:57.3713966Z ---------------------------------------------------------------------- 2022-03-17T21:50:57.9722180Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94230 2022-03-17T21:50:57.9759635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94231 2022-03-17T21:50:57.9761256Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:50:59.0983247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:50:59.0995312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:50:59.0997565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:59.1088904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:50:59.1096172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:50:59.2016290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:51:08.8988081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:08.8991641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:11.6086136Z ok (14.237s) 2022-03-17T21:51:11.6086633Z 2022-03-17T21:51:11.6087711Z ---------------------------------------------------------------------- 2022-03-17T21:51:11.6088729Z Ran 1 test in 14.237s 2022-03-17T21:51:11.6089204Z 2022-03-17T21:51:11.6089513Z OK 2022-03-17T21:51:11.6089916Z 2022-03-17T21:51:11.6090250Z Generating XML reports... 2022-03-17T21:51:11.6169953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215057.xml 2022-03-17T21:51:14.2889642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:51:14.2903068Z 2022-03-17T21:51:14.2903451Z Running tests... 2022-03-17T21:51:14.2904536Z ---------------------------------------------------------------------- 2022-03-17T21:51:14.9125586Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94444 2022-03-17T21:51:14.9164534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94445 2022-03-17T21:51:14.9165932Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:51:16.0914309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:51:16.0914876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:51:16.0915782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:51:16.0916619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:51:16.1021492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:51:16.1926142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:51:25.8327245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:25.8331163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:25.8359587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:25.8360937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:28.4435366Z ok (14.153s) 2022-03-17T21:51:28.4435796Z 2022-03-17T21:51:28.4436854Z ---------------------------------------------------------------------- 2022-03-17T21:51:28.4437768Z Ran 1 test in 14.153s 2022-03-17T21:51:28.4438166Z 2022-03-17T21:51:28.4438393Z OK 2022-03-17T21:51:28.4438691Z 2022-03-17T21:51:28.4439083Z Generating XML reports... 2022-03-17T21:51:28.4518486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215114.xml 2022-03-17T21:51:31.0572766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:51:31.0584921Z 2022-03-17T21:51:31.0636822Z Running tests... 2022-03-17T21:51:31.0638200Z ---------------------------------------------------------------------- 2022-03-17T21:51:31.6544056Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94658 2022-03-17T21:51:31.6577132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94659 2022-03-17T21:51:31.6578560Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:51:32.8102186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:51:32.8104214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:51:32.8105325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:51:32.8106790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:51:32.8120027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:51:32.9115958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:51:42.5507326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5509459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5540135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5541442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5821522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5822825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5841673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:42.5843280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:45.2898862Z ok (14.231s) 2022-03-17T21:51:45.2951028Z 2022-03-17T21:51:45.2951969Z ---------------------------------------------------------------------- 2022-03-17T21:51:45.2952832Z Ran 1 test in 14.231s 2022-03-17T21:51:45.2953214Z 2022-03-17T21:51:45.2953425Z OK 2022-03-17T21:51:45.2953698Z 2022-03-17T21:51:45.2954024Z Generating XML reports... 2022-03-17T21:51:45.2982201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215131.xml 2022-03-17T21:51:47.8371270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:51:47.8385873Z 2022-03-17T21:51:47.8386209Z Running tests... 2022-03-17T21:51:47.8386687Z ---------------------------------------------------------------------- 2022-03-17T21:51:48.4709431Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94872 2022-03-17T21:51:48.4745813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94873 2022-03-17T21:51:48.4747163Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:51:49.6159005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:51:49.6159555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:51:49.6160308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:51:49.6160978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:51:49.6267669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:51:49.7180439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:51:59.3069353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.3082680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.3126288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.3129452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.4021601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.4022886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.4032292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:51:59.4033528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:52:02.1011255Z ok (14.262s) 2022-03-17T21:52:02.1011679Z 2022-03-17T21:52:02.1012828Z ---------------------------------------------------------------------- 2022-03-17T21:52:02.1013665Z Ran 1 test in 14.262s 2022-03-17T21:52:02.1014190Z 2022-03-17T21:52:02.1014414Z OK 2022-03-17T21:52:02.1014743Z 2022-03-17T21:52:02.1015006Z Generating XML reports... 2022-03-17T21:52:02.1097453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215147.xml 2022-03-17T21:52:04.6776797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:52:04.6788206Z 2022-03-17T21:52:04.6788494Z Running tests... 2022-03-17T21:52:04.6788968Z ---------------------------------------------------------------------- 2022-03-17T21:52:05.2781212Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95086 2022-03-17T21:52:05.2812733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95087 2022-03-17T21:52:05.2813374Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:52:06.4103370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:52:06.4104167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:52:06.4105119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:06.4105959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:06.4108028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:52:06.4110973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:52:06.8371300Z 2022-03-17T21:52:08.5895375Z ok (3.910s) 2022-03-17T21:52:08.5895772Z 2022-03-17T21:52:08.5896948Z ---------------------------------------------------------------------- 2022-03-17T21:52:08.5897941Z Ran 1 test in 3.910s 2022-03-17T21:52:08.5898377Z 2022-03-17T21:52:08.5898625Z OK 2022-03-17T21:52:08.5898957Z 2022-03-17T21:52:08.5899307Z Generating XML reports... 2022-03-17T21:52:08.5977522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215204.xml 2022-03-17T21:52:11.1648005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:52:11.1658911Z 2022-03-17T21:52:11.1659169Z Running tests... 2022-03-17T21:52:11.1660358Z ---------------------------------------------------------------------- 2022-03-17T21:52:11.7662496Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95296 2022-03-17T21:52:11.7698455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95297 2022-03-17T21:52:11.7699802Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:52:12.9193143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:52:12.9194420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:52:12.9196395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:12.9198284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:12.9204869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:52:12.9206126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:52:14.8786521Z ok (3.712s) 2022-03-17T21:52:14.8786999Z 2022-03-17T21:52:14.8787959Z ---------------------------------------------------------------------- 2022-03-17T21:52:14.8788754Z Ran 1 test in 3.713s 2022-03-17T21:52:14.8789130Z 2022-03-17T21:52:14.8789327Z OK 2022-03-17T21:52:14.8795400Z 2022-03-17T21:52:14.8796099Z Generating XML reports... 2022-03-17T21:52:14.8864306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215211.xml 2022-03-17T21:52:17.4346066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:52:17.4358789Z 2022-03-17T21:52:17.4359346Z Running tests... 2022-03-17T21:52:17.4360520Z ---------------------------------------------------------------------- 2022-03-17T21:52:18.0421314Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95506 2022-03-17T21:52:18.0467684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95507 2022-03-17T21:52:18.0468995Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:52:19.1703956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:52:19.1706515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:19.1707920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:52:19.1710068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:19.1812497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:52:19.2725815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:52:20.9564371Z skip (3.520s) 2022-03-17T21:52:20.9564962Z 2022-03-17T21:52:20.9566188Z ---------------------------------------------------------------------- 2022-03-17T21:52:20.9567184Z Ran 1 test in 3.520s 2022-03-17T21:52:20.9567653Z 2022-03-17T21:52:20.9567906Z OK (skipped=1) 2022-03-17T21:52:20.9568232Z 2022-03-17T21:52:20.9568527Z Generating XML reports... 2022-03-17T21:52:20.9645857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215217.xml 2022-03-17T21:52:23.5956284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:52:23.5969944Z 2022-03-17T21:52:23.5970337Z Running tests... 2022-03-17T21:52:23.5971599Z ---------------------------------------------------------------------- 2022-03-17T21:52:24.2197119Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95716 2022-03-17T21:52:24.2236974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95717 2022-03-17T21:52:24.2238311Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:52:25.3706167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:52:25.3713545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:52:25.3715763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:25.3810686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:25.3812671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:52:25.4735409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:52:35.2517250Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:52:38.0523173Z ok (14.455s) 2022-03-17T21:52:38.0523720Z 2022-03-17T21:52:38.0524972Z ---------------------------------------------------------------------- 2022-03-17T21:52:38.0525876Z Ran 1 test in 14.455s 2022-03-17T21:52:38.0526276Z 2022-03-17T21:52:38.0526499Z OK 2022-03-17T21:52:38.0526825Z 2022-03-17T21:52:38.0527090Z Generating XML reports... 2022-03-17T21:52:38.0609748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215223.xml 2022-03-17T21:52:40.6548044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:52:40.6562679Z 2022-03-17T21:52:40.6562912Z Running tests... 2022-03-17T21:52:40.6563650Z ---------------------------------------------------------------------- 2022-03-17T21:52:41.2827180Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95930 2022-03-17T21:52:41.2861772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95931 2022-03-17T21:52:41.2862519Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:52:42.4457090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:52:42.4458428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:52:42.4460418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:42.4462294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:42.4463650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:52:42.4464809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:52:52.0722330Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:52:52.0727246Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:52:54.8164530Z ok (14.160s) 2022-03-17T21:52:54.8164995Z 2022-03-17T21:52:54.8165890Z ---------------------------------------------------------------------- 2022-03-17T21:52:54.8166966Z Ran 1 test in 14.160s 2022-03-17T21:52:54.8167343Z 2022-03-17T21:52:54.8167570Z OK 2022-03-17T21:52:54.8167848Z 2022-03-17T21:52:54.8168171Z Generating XML reports... 2022-03-17T21:52:54.8248720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215240.xml 2022-03-17T21:52:57.3876659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:52:57.3887725Z 2022-03-17T21:52:57.3888039Z Running tests... 2022-03-17T21:52:57.3889148Z ---------------------------------------------------------------------- 2022-03-17T21:52:58.0054906Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96144 2022-03-17T21:52:58.0102340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96145 2022-03-17T21:52:58.0103657Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:52:59.1853372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:52:59.1855700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:59.1857137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:52:59.1858952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:52:59.1865398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:52:59.2862700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:52:59.2893280Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2900310Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2908643Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:989.) 2022-03-17T21:52:59.2911316Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T21:52:59.2915667Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:989.) 2022-03-17T21:52:59.2918270Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T21:52:59.2919487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:52:59.2920739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:52:59.2924074Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2928120Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2932088Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2936084Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2940295Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2944438Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2948430Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2952349Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2956682Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:52:59.2960621Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:53:01.0202496Z ok (3.631s) 2022-03-17T21:53:01.0203029Z 2022-03-17T21:53:01.0204732Z ---------------------------------------------------------------------- 2022-03-17T21:53:01.0205572Z Ran 1 test in 3.631s 2022-03-17T21:53:01.0206008Z 2022-03-17T21:53:01.0206264Z OK 2022-03-17T21:53:01.0206641Z 2022-03-17T21:53:01.0206977Z Generating XML reports... 2022-03-17T21:53:01.0284077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215257.xml 2022-03-17T21:53:03.6352749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:53:03.6366253Z 2022-03-17T21:53:03.6367153Z Running tests... 2022-03-17T21:53:03.6368408Z ---------------------------------------------------------------------- 2022-03-17T21:53:04.2466061Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96358 2022-03-17T21:53:04.2507671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96359 2022-03-17T21:53:04.2509107Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:53:05.4021701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:53:05.4022508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:53:05.4023031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:53:05.4023974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:53:05.4039815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:53:05.5040504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:53:15.0787028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:53:15.0794192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:53:17.8791486Z ok (14.242s) 2022-03-17T21:53:17.8792145Z 2022-03-17T21:53:17.8793282Z ---------------------------------------------------------------------- 2022-03-17T21:53:17.8794108Z Ran 1 test in 14.242s 2022-03-17T21:53:17.8794552Z 2022-03-17T21:53:17.8796919Z OK 2022-03-17T21:53:17.8797360Z 2022-03-17T21:53:17.8797702Z Generating XML reports... 2022-03-17T21:53:17.8866319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215303.xml 2022-03-17T21:53:20.4315133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:53:20.4326488Z 2022-03-17T21:53:20.4326828Z Running tests... 2022-03-17T21:53:20.4327279Z ---------------------------------------------------------------------- 2022-03-17T21:53:21.0343869Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96572 2022-03-17T21:53:21.0389588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96573 2022-03-17T21:53:21.0391535Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:53:22.1797858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:53:22.1798740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:53:22.1799540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:53:22.1801116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:53:22.1818070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:53:22.2814943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:53:22.6870168Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:53:22.6872538Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:53:22.6906981Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:53:22.6909323Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:53:34.5667404Z ok (14.134s) 2022-03-17T21:53:34.5668453Z 2022-03-17T21:53:34.5669883Z ---------------------------------------------------------------------- 2022-03-17T21:53:34.5670757Z Ran 1 test in 14.134s 2022-03-17T21:53:34.5671154Z 2022-03-17T21:53:34.5671383Z OK 2022-03-17T21:53:34.5671707Z 2022-03-17T21:53:34.5671974Z Generating XML reports... 2022-03-17T21:53:34.5750657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215320.xml 2022-03-17T21:53:37.1253584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:53:37.1263422Z 2022-03-17T21:53:37.1263834Z Running tests... 2022-03-17T21:53:37.1265045Z ---------------------------------------------------------------------- 2022-03-17T21:53:37.7260935Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96848 2022-03-17T21:53:37.7298989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96849 2022-03-17T21:53:37.7300399Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:53:38.8766656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:53:38.8768796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:53:38.8770246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:53:38.8772065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:53:38.8876134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:53:38.9789830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:53:50.8940909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:53:50.8942366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:53:56.1655972Z ok (19.039s) 2022-03-17T21:53:56.1656489Z 2022-03-17T21:53:56.1657503Z ---------------------------------------------------------------------- 2022-03-17T21:53:56.1658479Z Ran 1 test in 19.039s 2022-03-17T21:53:56.1662083Z 2022-03-17T21:53:56.1662517Z OK 2022-03-17T21:53:56.1662944Z 2022-03-17T21:53:56.1663282Z Generating XML reports... 2022-03-17T21:53:56.1729904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215337.xml 2022-03-17T21:53:58.8043824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:53:58.8056688Z 2022-03-17T21:53:58.8057047Z Running tests... 2022-03-17T21:53:58.8057496Z ---------------------------------------------------------------------- 2022-03-17T21:53:59.4078125Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97062 2022-03-17T21:53:59.4113229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97063 2022-03-17T21:53:59.4116018Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:00.5452038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:00.5452778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:00.5453717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:00.5454533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:00.5464921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:00.5466612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:10.1597148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:54:10.1598475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:54:12.8435482Z ok (14.037s) 2022-03-17T21:54:12.8436141Z 2022-03-17T21:54:12.8437326Z ---------------------------------------------------------------------- 2022-03-17T21:54:12.8438268Z Ran 1 test in 14.038s 2022-03-17T21:54:12.8438742Z 2022-03-17T21:54:12.8439022Z OK 2022-03-17T21:54:12.8439512Z 2022-03-17T21:54:12.8439822Z Generating XML reports... 2022-03-17T21:54:12.8505814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215358.xml 2022-03-17T21:54:15.3912271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:15.3925150Z 2022-03-17T21:54:15.3925436Z Running tests... 2022-03-17T21:54:15.3925904Z ---------------------------------------------------------------------- 2022-03-17T21:54:15.9916604Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97276 2022-03-17T21:54:15.9957548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97277 2022-03-17T21:54:15.9958937Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:17.1502501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:17.1503724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:17.1505654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:17.1507957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:17.1521078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:17.2518634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:18.8047380Z skip (3.412s) 2022-03-17T21:54:18.8048056Z 2022-03-17T21:54:18.8049270Z ---------------------------------------------------------------------- 2022-03-17T21:54:18.8050396Z Ran 1 test in 3.412s 2022-03-17T21:54:18.8050960Z 2022-03-17T21:54:18.8051234Z OK (skipped=1) 2022-03-17T21:54:18.8051842Z 2022-03-17T21:54:18.8052131Z Generating XML reports... 2022-03-17T21:54:18.8131768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215415.xml 2022-03-17T21:54:21.4113627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:21.4126156Z 2022-03-17T21:54:21.4126767Z Running tests... 2022-03-17T21:54:21.4127940Z ---------------------------------------------------------------------- 2022-03-17T21:54:22.0086416Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97486 2022-03-17T21:54:22.0118271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97487 2022-03-17T21:54:22.0119649Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:23.1270789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:23.1272727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:23.1274094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:23.1275773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:23.1277131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:23.1278665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:24.9214486Z skip (3.509s) 2022-03-17T21:54:24.9215190Z 2022-03-17T21:54:24.9216264Z ---------------------------------------------------------------------- 2022-03-17T21:54:24.9217548Z Ran 1 test in 3.509s 2022-03-17T21:54:24.9217911Z 2022-03-17T21:54:24.9218219Z OK (skipped=1) 2022-03-17T21:54:24.9218595Z 2022-03-17T21:54:24.9218843Z Generating XML reports... 2022-03-17T21:54:24.9299393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215421.xml 2022-03-17T21:54:27.4912408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:27.4924581Z 2022-03-17T21:54:27.4924735Z Running tests... 2022-03-17T21:54:27.4925185Z ---------------------------------------------------------------------- 2022-03-17T21:54:28.0909628Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97696 2022-03-17T21:54:28.0941900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97697 2022-03-17T21:54:28.0942396Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:29.2598986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:29.2599795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:29.2600650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:29.2601427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:29.2707029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:29.3626239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:31.0025744Z skip (3.509s) 2022-03-17T21:54:31.0026150Z 2022-03-17T21:54:31.0027151Z ---------------------------------------------------------------------- 2022-03-17T21:54:31.0027910Z Ran 1 test in 3.510s 2022-03-17T21:54:31.0028280Z 2022-03-17T21:54:31.0028506Z OK (skipped=1) 2022-03-17T21:54:31.0028846Z 2022-03-17T21:54:31.0029117Z Generating XML reports... 2022-03-17T21:54:31.0106234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215427.xml 2022-03-17T21:54:33.5907328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:33.5918855Z 2022-03-17T21:54:33.5919103Z Running tests... 2022-03-17T21:54:33.5919538Z ---------------------------------------------------------------------- 2022-03-17T21:54:34.1919589Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97906 2022-03-17T21:54:34.1957925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97907 2022-03-17T21:54:34.1959156Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:35.3564643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:35.3565133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:35.3565855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:35.3566673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:35.3571602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:35.3572493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:37.1053410Z skip (3.513s) 2022-03-17T21:54:37.1053822Z 2022-03-17T21:54:37.1054885Z ---------------------------------------------------------------------- 2022-03-17T21:54:37.1055852Z Ran 1 test in 3.513s 2022-03-17T21:54:37.1056312Z 2022-03-17T21:54:37.1056609Z OK (skipped=1) 2022-03-17T21:54:37.1056947Z 2022-03-17T21:54:37.1057230Z Generating XML reports... 2022-03-17T21:54:37.1137010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215433.xml 2022-03-17T21:54:39.7162913Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:39.7174315Z 2022-03-17T21:54:39.7174581Z Running tests... 2022-03-17T21:54:39.7176348Z ---------------------------------------------------------------------- 2022-03-17T21:54:40.3576047Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98116 2022-03-17T21:54:40.3627400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98117 2022-03-17T21:54:40.3628716Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:41.5252267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:41.5253378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:41.5254242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:41.5357056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:41.5366611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:41.6366491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:43.9729596Z skip (4.255s) 2022-03-17T21:54:43.9729994Z 2022-03-17T21:54:43.9730943Z ---------------------------------------------------------------------- 2022-03-17T21:54:43.9732056Z Ran 1 test in 4.255s 2022-03-17T21:54:43.9732424Z 2022-03-17T21:54:43.9732668Z OK (skipped=1) 2022-03-17T21:54:43.9733030Z 2022-03-17T21:54:43.9733291Z Generating XML reports... 2022-03-17T21:54:43.9817119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215439.xml 2022-03-17T21:54:46.5771035Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:46.5782507Z 2022-03-17T21:54:46.5782779Z Running tests... 2022-03-17T21:54:46.5783236Z ---------------------------------------------------------------------- 2022-03-17T21:54:47.1728821Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98326 2022-03-17T21:54:47.1762039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98327 2022-03-17T21:54:47.1763395Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:48.3216266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:48.3217464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:48.3219591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:48.3224657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:48.3333435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:48.4238392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:50.1838679Z skip (3.605s) 2022-03-17T21:54:50.1839082Z 2022-03-17T21:54:50.1840025Z ---------------------------------------------------------------------- 2022-03-17T21:54:50.1840805Z Ran 1 test in 3.605s 2022-03-17T21:54:50.1841193Z 2022-03-17T21:54:50.1841420Z OK (skipped=1) 2022-03-17T21:54:50.1841752Z 2022-03-17T21:54:50.1842498Z Generating XML reports... 2022-03-17T21:54:50.1922663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215446.xml 2022-03-17T21:54:52.8033265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:52.8047307Z 2022-03-17T21:54:52.8048013Z Running tests... 2022-03-17T21:54:52.8049219Z ---------------------------------------------------------------------- 2022-03-17T21:54:53.4313732Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98536 2022-03-17T21:54:53.4349095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98537 2022-03-17T21:54:53.4350437Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:54:54.5725423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:54:54.5727538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:54.5729042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:54:54.5730741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:54:54.5830871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:54:54.6741771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:54:56.3445976Z skip (3.539s) 2022-03-17T21:54:56.3446467Z 2022-03-17T21:54:56.3447475Z ---------------------------------------------------------------------- 2022-03-17T21:54:56.3448489Z Ran 1 test in 3.540s 2022-03-17T21:54:56.3448946Z 2022-03-17T21:54:56.3450055Z OK (skipped=1) 2022-03-17T21:54:56.3450487Z 2022-03-17T21:54:56.3450824Z Generating XML reports... 2022-03-17T21:54:56.3533700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215452.xml 2022-03-17T21:54:58.9951492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:54:58.9964825Z 2022-03-17T21:54:58.9965462Z Running tests... 2022-03-17T21:54:58.9966859Z ---------------------------------------------------------------------- 2022-03-17T21:54:59.6325310Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98746 2022-03-17T21:54:59.6364559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98747 2022-03-17T21:54:59.6365950Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:00.7898006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:00.7899830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:00.7901337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:00.7902798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:00.7912335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:00.8911562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:02.5458671Z skip (3.549s) 2022-03-17T21:55:02.5459157Z 2022-03-17T21:55:02.5460187Z ---------------------------------------------------------------------- 2022-03-17T21:55:02.5461086Z Ran 1 test in 3.550s 2022-03-17T21:55:02.5461512Z 2022-03-17T21:55:02.5461769Z OK (skipped=1) 2022-03-17T21:55:02.5462136Z 2022-03-17T21:55:02.5462465Z Generating XML reports... 2022-03-17T21:55:02.5542449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215458.xml 2022-03-17T21:55:05.1293687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:05.1305038Z 2022-03-17T21:55:05.1305370Z Running tests... 2022-03-17T21:55:05.1306545Z ---------------------------------------------------------------------- 2022-03-17T21:55:05.7303282Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98956 2022-03-17T21:55:05.7337535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98957 2022-03-17T21:55:05.7338833Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:06.8621518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:06.8622737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:06.8624574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:06.8626321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:06.8628500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:06.8630844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:08.5423764Z skip (3.411s) 2022-03-17T21:55:08.5424172Z 2022-03-17T21:55:08.5425258Z ---------------------------------------------------------------------- 2022-03-17T21:55:08.5426041Z Ran 1 test in 3.412s 2022-03-17T21:55:08.5426412Z 2022-03-17T21:55:08.5426639Z OK (skipped=1) 2022-03-17T21:55:08.5426987Z 2022-03-17T21:55:08.5427257Z Generating XML reports... 2022-03-17T21:55:08.5499718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215505.xml 2022-03-17T21:55:11.1541745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:11.1553007Z 2022-03-17T21:55:11.1553864Z Running tests... 2022-03-17T21:55:11.1554683Z ---------------------------------------------------------------------- 2022-03-17T21:55:11.7910905Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99166 2022-03-17T21:55:11.7949819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99167 2022-03-17T21:55:11.7951219Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:12.9082887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:12.9083500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:12.9084552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:12.9085376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:12.9198715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:13.0094358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:14.8027838Z skip (3.647s) 2022-03-17T21:55:14.8028259Z 2022-03-17T21:55:14.8029296Z ---------------------------------------------------------------------- 2022-03-17T21:55:14.8030062Z Ran 1 test in 3.647s 2022-03-17T21:55:14.8085560Z 2022-03-17T21:55:14.8086660Z OK (skipped=1) 2022-03-17T21:55:14.8087114Z 2022-03-17T21:55:14.8087403Z Generating XML reports... 2022-03-17T21:55:14.8098855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215511.xml 2022-03-17T21:55:17.3845447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:17.3856505Z 2022-03-17T21:55:17.3857064Z Running tests... 2022-03-17T21:55:17.3858267Z ---------------------------------------------------------------------- 2022-03-17T21:55:18.0089756Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99376 2022-03-17T21:55:18.0130060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99377 2022-03-17T21:55:18.0131392Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:19.1122986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:19.1125156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:19.1126483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:19.1128179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:19.1232594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:19.2149680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:20.9226006Z skip (3.536s) 2022-03-17T21:55:20.9226663Z 2022-03-17T21:55:20.9227839Z ---------------------------------------------------------------------- 2022-03-17T21:55:20.9228714Z Ran 1 test in 3.537s 2022-03-17T21:55:20.9229072Z 2022-03-17T21:55:20.9229301Z OK (skipped=1) 2022-03-17T21:55:20.9229632Z 2022-03-17T21:55:20.9229899Z Generating XML reports... 2022-03-17T21:55:20.9305499Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215517.xml 2022-03-17T21:55:23.4803089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:23.4811553Z 2022-03-17T21:55:23.4811908Z Running tests... 2022-03-17T21:55:23.4813340Z ---------------------------------------------------------------------- 2022-03-17T21:55:24.0809580Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99586 2022-03-17T21:55:24.0875415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99587 2022-03-17T21:55:24.0876821Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:25.2317198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:25.2318509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:25.2320425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:25.2322450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:25.2431671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:25.3334031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:26.8957225Z skip (3.414s) 2022-03-17T21:55:26.8957618Z 2022-03-17T21:55:26.8958532Z ---------------------------------------------------------------------- 2022-03-17T21:55:26.8959324Z Ran 1 test in 3.414s 2022-03-17T21:55:26.8959696Z 2022-03-17T21:55:26.8959930Z OK (skipped=1) 2022-03-17T21:55:26.8960280Z 2022-03-17T21:55:26.8960550Z Generating XML reports... 2022-03-17T21:55:26.9041024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215523.xml 2022-03-17T21:55:29.5213062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:29.5226414Z 2022-03-17T21:55:29.5227140Z Running tests... 2022-03-17T21:55:29.5228411Z ---------------------------------------------------------------------- 2022-03-17T21:55:30.1448523Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99796 2022-03-17T21:55:30.1487697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99797 2022-03-17T21:55:30.1489135Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:31.3058797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:31.3060434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:31.3062640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:31.3157905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:31.3161463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:31.4073249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:33.0583679Z skip (3.535s) 2022-03-17T21:55:33.0584072Z 2022-03-17T21:55:33.0585037Z ---------------------------------------------------------------------- 2022-03-17T21:55:33.0585850Z Ran 1 test in 3.536s 2022-03-17T21:55:33.0586224Z 2022-03-17T21:55:33.0586437Z OK (skipped=1) 2022-03-17T21:55:33.0586786Z 2022-03-17T21:55:33.0587054Z Generating XML reports... 2022-03-17T21:55:33.0663829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215529.xml 2022-03-17T21:55:35.6868604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:35.6876416Z 2022-03-17T21:55:35.6877162Z Running tests... 2022-03-17T21:55:35.6878539Z ---------------------------------------------------------------------- 2022-03-17T21:55:36.2844031Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100006 2022-03-17T21:55:36.2886476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100007 2022-03-17T21:55:36.2887837Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:37.4647746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:37.4649207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:37.4651288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:37.4749011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:37.4770504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:37.5761676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:39.1971146Z skip (3.509s) 2022-03-17T21:55:39.1972197Z 2022-03-17T21:55:39.1973944Z ---------------------------------------------------------------------- 2022-03-17T21:55:39.1974832Z Ran 1 test in 3.509s 2022-03-17T21:55:39.1975214Z 2022-03-17T21:55:39.1975466Z OK (skipped=1) 2022-03-17T21:55:39.1975817Z 2022-03-17T21:55:39.1976710Z Generating XML reports... 2022-03-17T21:55:39.2055217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215535.xml 2022-03-17T21:55:41.8114553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:41.8124169Z 2022-03-17T21:55:41.8124488Z Running tests... 2022-03-17T21:55:41.8125039Z ---------------------------------------------------------------------- 2022-03-17T21:55:42.4112036Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100216 2022-03-17T21:55:42.4143806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100217 2022-03-17T21:55:42.4145189Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:43.5229115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:43.5229719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:43.5230596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:43.5231416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:43.5235047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:43.5235572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:45.8247159Z skip (4.012s) 2022-03-17T21:55:45.8247560Z 2022-03-17T21:55:45.8248391Z ---------------------------------------------------------------------- 2022-03-17T21:55:45.8249176Z Ran 1 test in 4.012s 2022-03-17T21:55:45.8249558Z 2022-03-17T21:55:45.8249815Z OK (skipped=1) 2022-03-17T21:55:45.8250141Z 2022-03-17T21:55:45.8250424Z Generating XML reports... 2022-03-17T21:55:45.8328607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215541.xml 2022-03-17T21:55:48.3972651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:48.3986189Z 2022-03-17T21:55:48.3986509Z Running tests... 2022-03-17T21:55:48.3987289Z ---------------------------------------------------------------------- 2022-03-17T21:55:48.9959724Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100426 2022-03-17T21:55:48.9995803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100427 2022-03-17T21:55:48.9997715Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:50.1512609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:50.1513501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:50.1514103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:50.1514876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:50.1519461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:55:50.1520005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:52.0088737Z skip (3.610s) 2022-03-17T21:55:52.0089172Z 2022-03-17T21:55:52.0090026Z ---------------------------------------------------------------------- 2022-03-17T21:55:52.0091306Z Ran 1 test in 3.610s 2022-03-17T21:55:52.0091684Z 2022-03-17T21:55:52.0095604Z OK (skipped=1) 2022-03-17T21:55:52.0096058Z 2022-03-17T21:55:52.0096354Z Generating XML reports... 2022-03-17T21:55:52.0172520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215548.xml 2022-03-17T21:55:54.5967171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:55:54.5983349Z 2022-03-17T21:55:54.5983896Z Running tests... 2022-03-17T21:55:54.5985093Z ---------------------------------------------------------------------- 2022-03-17T21:55:55.2254200Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100636 2022-03-17T21:55:55.2291930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100637 2022-03-17T21:55:55.2293259Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:55:56.3795888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:55:56.3797226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:55:56.3799285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:56.3801181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:55:56.3907655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:55:56.4822281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:56:06.1965760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.1978355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.2302091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.2303196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.2495630Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:56:06.2500397Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:56:06.3042192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3043391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3274245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3275463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3438021Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:56:06.3440531Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:56:06.3443817Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:56:06.3445935Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:56:06.3520656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3522065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3755856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:06.3757089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:09.0622722Z ok (14.464s) 2022-03-17T21:56:09.0623226Z 2022-03-17T21:56:09.0624372Z ---------------------------------------------------------------------- 2022-03-17T21:56:09.0625333Z Ran 1 test in 14.464s 2022-03-17T21:56:09.0625785Z 2022-03-17T21:56:09.0625985Z OK 2022-03-17T21:56:09.0626337Z 2022-03-17T21:56:09.0626661Z Generating XML reports... 2022-03-17T21:56:09.0700221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215554.xml 2022-03-17T21:56:11.6845776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:56:11.6857344Z 2022-03-17T21:56:11.6857586Z Running tests... 2022-03-17T21:56:11.6858031Z ---------------------------------------------------------------------- 2022-03-17T21:56:12.2874583Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100850 2022-03-17T21:56:12.2909180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100851 2022-03-17T21:56:12.2910881Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:56:13.4435562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:56:13.4437648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:13.4438952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:56:13.4440627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:13.4542982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:56:13.5457526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:56:25.7148636Z ok (14.028s) 2022-03-17T21:56:25.7149064Z 2022-03-17T21:56:25.7150033Z ---------------------------------------------------------------------- 2022-03-17T21:56:25.7150860Z Ran 1 test in 14.029s 2022-03-17T21:56:25.7151231Z 2022-03-17T21:56:25.7151422Z OK 2022-03-17T21:56:25.7151705Z 2022-03-17T21:56:25.7151990Z Generating XML reports... 2022-03-17T21:56:25.7226131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215611.xml 2022-03-17T21:56:28.3601132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:56:28.3609229Z 2022-03-17T21:56:28.3609988Z Running tests... 2022-03-17T21:56:28.3611120Z ---------------------------------------------------------------------- 2022-03-17T21:56:28.9607242Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101060 2022-03-17T21:56:28.9641419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101061 2022-03-17T21:56:28.9643124Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:56:30.1124461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:56:30.1136482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:56:30.1137667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:30.1223118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:30.1226514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:56:30.2155292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:56:39.6616346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:39.6617627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:42.2854282Z ok (13.924s) 2022-03-17T21:56:42.2854663Z 2022-03-17T21:56:42.2856011Z ---------------------------------------------------------------------- 2022-03-17T21:56:42.2856806Z Ran 1 test in 13.924s 2022-03-17T21:56:42.2857172Z 2022-03-17T21:56:42.2857376Z OK 2022-03-17T21:56:42.2857668Z 2022-03-17T21:56:42.2857946Z Generating XML reports... 2022-03-17T21:56:42.2939144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215628.xml 2022-03-17T21:56:44.9015304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:56:44.9029874Z 2022-03-17T21:56:44.9030174Z Running tests... 2022-03-17T21:56:44.9030711Z ---------------------------------------------------------------------- 2022-03-17T21:56:45.5352418Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101274 2022-03-17T21:56:45.5388940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101275 2022-03-17T21:56:45.5389561Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:56:46.7086990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:56:46.7087770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:46.7088318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:56:46.7089054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:46.7199928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:56:46.8111056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:56:46.8375122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:46.8376520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:56:48.7482164Z ok (3.845s) 2022-03-17T21:56:48.7482749Z 2022-03-17T21:56:48.7483801Z ---------------------------------------------------------------------- 2022-03-17T21:56:48.7485096Z Ran 1 test in 3.845s 2022-03-17T21:56:48.7485465Z 2022-03-17T21:56:48.7485655Z OK 2022-03-17T21:56:48.7485969Z 2022-03-17T21:56:48.7486245Z Generating XML reports... 2022-03-17T21:56:48.7566190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215644.xml 2022-03-17T21:56:51.3864894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:56:51.3876340Z 2022-03-17T21:56:51.3876891Z Running tests... 2022-03-17T21:56:51.3878055Z ---------------------------------------------------------------------- 2022-03-17T21:56:51.9865002Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101550 2022-03-17T21:56:51.9897401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101551 2022-03-17T21:56:51.9899372Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:56:53.1239603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:56:53.1240502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:53.1241081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:56:53.1241944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:56:53.1250995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:56:53.2250411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:57:03.0091955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:57:03.0093751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:57:05.8234119Z ok (14.435s) 2022-03-17T21:57:05.8234497Z 2022-03-17T21:57:05.8236358Z ---------------------------------------------------------------------- 2022-03-17T21:57:05.8237188Z Ran 1 test in 14.435s 2022-03-17T21:57:05.8237555Z 2022-03-17T21:57:05.8237743Z OK 2022-03-17T21:57:05.8240004Z 2022-03-17T21:57:05.8240538Z Generating XML reports... 2022-03-17T21:57:05.8304313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215651.xml 2022-03-17T21:57:08.5233136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:57:08.5242116Z 2022-03-17T21:57:08.5242416Z Running tests... 2022-03-17T21:57:08.5243785Z ---------------------------------------------------------------------- 2022-03-17T21:57:09.1671777Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101764 2022-03-17T21:57:09.1715080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101765 2022-03-17T21:57:09.1716643Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:57:10.3108616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:57:10.3110973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:10.3112320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:57:10.3114034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:10.3122511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:57:10.4121173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:57:12.1809061Z skip (3.656s) 2022-03-17T21:57:12.1809613Z 2022-03-17T21:57:12.1810632Z ---------------------------------------------------------------------- 2022-03-17T21:57:12.1811618Z Ran 1 test in 3.656s 2022-03-17T21:57:12.1812093Z 2022-03-17T21:57:12.1812366Z OK (skipped=1) 2022-03-17T21:57:12.1812917Z 2022-03-17T21:57:12.1814186Z Generating XML reports... 2022-03-17T21:57:12.1893628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215708.xml 2022-03-17T21:57:14.7326213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:57:14.7336878Z 2022-03-17T21:57:14.7337449Z Running tests... 2022-03-17T21:57:14.7338629Z ---------------------------------------------------------------------- 2022-03-17T21:57:15.3325043Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101974 2022-03-17T21:57:15.3363009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101975 2022-03-17T21:57:15.3364383Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:57:16.4895676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:57:16.4897713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:16.4899080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:57:16.4900786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:16.5005173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:57:16.5919836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:57:18.3447378Z skip (3.610s) 2022-03-17T21:57:18.3448040Z 2022-03-17T21:57:18.3449296Z ---------------------------------------------------------------------- 2022-03-17T21:57:18.3450258Z Ran 1 test in 3.611s 2022-03-17T21:57:18.3450746Z 2022-03-17T21:57:18.3451004Z OK (skipped=1) 2022-03-17T21:57:18.3451364Z 2022-03-17T21:57:18.3452265Z Generating XML reports... 2022-03-17T21:57:18.3532292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215714.xml 2022-03-17T21:57:20.9705771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:57:20.9715909Z 2022-03-17T21:57:20.9716636Z Running tests... 2022-03-17T21:57:20.9718106Z ---------------------------------------------------------------------- 2022-03-17T21:57:21.6022254Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102184 2022-03-17T21:57:21.6058354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102185 2022-03-17T21:57:21.6059956Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:57:22.7627693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:57:22.7629206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:57:22.7631326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:22.7633194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:22.7734304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:57:22.8645762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:57:35.3406099Z ok (14.368s) 2022-03-17T21:57:35.3406604Z 2022-03-17T21:57:35.3407794Z ---------------------------------------------------------------------- 2022-03-17T21:57:35.3408822Z Ran 1 test in 14.369s 2022-03-17T21:57:35.3409460Z 2022-03-17T21:57:35.3409729Z OK 2022-03-17T21:57:35.3410050Z 2022-03-17T21:57:35.3410335Z Generating XML reports... 2022-03-17T21:57:35.3474001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215720.xml 2022-03-17T21:57:37.9141445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:57:37.9150549Z 2022-03-17T21:57:37.9151008Z Running tests... 2022-03-17T21:57:37.9152091Z ---------------------------------------------------------------------- 2022-03-17T21:57:38.5153245Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102406 2022-03-17T21:57:38.5194089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102407 2022-03-17T21:57:38.5195463Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:57:39.6912474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:57:39.6914812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:39.6916272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:57:39.6918167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:39.6934587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:57:39.7930490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:57:52.4440506Z ok (14.528s) 2022-03-17T21:57:52.4440983Z 2022-03-17T21:57:52.4442525Z ---------------------------------------------------------------------- 2022-03-17T21:57:52.4443970Z Ran 1 test in 14.529s 2022-03-17T21:57:52.4444344Z 2022-03-17T21:57:52.4444558Z OK 2022-03-17T21:57:52.4444883Z 2022-03-17T21:57:52.4445169Z Generating XML reports... 2022-03-17T21:57:52.4520749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215737.xml 2022-03-17T21:57:55.0012967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:57:55.0024348Z 2022-03-17T21:57:55.0024586Z Running tests... 2022-03-17T21:57:55.0025183Z ---------------------------------------------------------------------- 2022-03-17T21:57:55.6006561Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102628 2022-03-17T21:57:55.6042517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102629 2022-03-17T21:57:55.6044161Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:57:56.7537845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:57:56.7539107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:57:56.7541132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:56.7542912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:57:56.7554933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:57:56.7556141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:58:08.9249479Z ok (13.922s) 2022-03-17T21:58:08.9249864Z 2022-03-17T21:58:08.9250922Z ---------------------------------------------------------------------- 2022-03-17T21:58:08.9251804Z Ran 1 test in 13.922s 2022-03-17T21:58:08.9252193Z 2022-03-17T21:58:08.9252369Z OK 2022-03-17T21:58:08.9252677Z 2022-03-17T21:58:08.9252962Z Generating XML reports... 2022-03-17T21:58:08.9329391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215754.xml 2022-03-17T21:58:11.5653392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:58:11.5665591Z 2022-03-17T21:58:11.5665752Z Running tests... 2022-03-17T21:58:11.5666303Z ---------------------------------------------------------------------- 2022-03-17T21:58:12.1662862Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102838 2022-03-17T21:58:12.1691777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102839 2022-03-17T21:58:12.1693147Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:58:13.3040428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:58:13.3041014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:58:13.3042045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:58:13.3043295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:58:13.3154872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:58:13.4063813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:58:22.8286062Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:58:22.8598974Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:58:23.0173730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:58:23.0177025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:58:25.6955374Z ok (14.128s) 2022-03-17T21:58:25.6955771Z 2022-03-17T21:58:25.6956823Z ---------------------------------------------------------------------- 2022-03-17T21:58:25.6957585Z Ran 1 test in 14.129s 2022-03-17T21:58:25.6957951Z 2022-03-17T21:58:25.6958140Z OK 2022-03-17T21:58:25.6958462Z 2022-03-17T21:58:25.6958744Z Generating XML reports... 2022-03-17T21:58:25.7030453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215811.xml 2022-03-17T21:58:28.3806180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:58:28.3818509Z 2022-03-17T21:58:28.3818789Z Running tests... 2022-03-17T21:58:28.3819917Z ---------------------------------------------------------------------- 2022-03-17T21:58:29.0213284Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103052 2022-03-17T21:58:29.0254243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103053 2022-03-17T21:58:29.0255595Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:58:30.1651643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:58:30.1703106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:58:30.1703639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:58:30.1704382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:58:30.1762182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:58:30.2676322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:58:30.6746377Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:58:30.6748585Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:58:30.6752210Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:58:30.6754319Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:58:42.5516877Z ok (14.169s) 2022-03-17T21:58:42.5517325Z 2022-03-17T21:58:42.5518259Z ---------------------------------------------------------------------- 2022-03-17T21:58:42.5519087Z Ran 1 test in 14.170s 2022-03-17T21:58:42.5519459Z 2022-03-17T21:58:42.5519638Z OK 2022-03-17T21:58:42.5519957Z 2022-03-17T21:58:42.5520239Z Generating XML reports... 2022-03-17T21:58:42.5600502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215828.xml 2022-03-17T21:58:45.1689113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:58:45.1699411Z 2022-03-17T21:58:45.1699555Z Running tests... 2022-03-17T21:58:45.1700111Z ---------------------------------------------------------------------- 2022-03-17T21:58:45.7677520Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103266 2022-03-17T21:58:45.7713005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103267 2022-03-17T21:58:45.7714366Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:58:46.8856328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:58:46.8857507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:58:46.8858082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:58:46.8858837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:58:46.8967537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:58:46.9880583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:58:56.5342150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:58:56.5347894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:58:56.5955579Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:58:56.5959906Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:58:59.2022201Z ok (14.032s) 2022-03-17T21:58:59.2022771Z 2022-03-17T21:58:59.2024039Z ---------------------------------------------------------------------- 2022-03-17T21:58:59.2025014Z Ran 1 test in 14.032s 2022-03-17T21:58:59.2025381Z 2022-03-17T21:58:59.2025554Z OK 2022-03-17T21:58:59.2025863Z 2022-03-17T21:58:59.2029971Z Generating XML reports... 2022-03-17T21:58:59.2104973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215845.xml 2022-03-17T21:59:01.7765581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:59:01.7778628Z 2022-03-17T21:59:01.7778753Z Running tests... 2022-03-17T21:59:01.7779204Z ---------------------------------------------------------------------- 2022-03-17T21:59:02.3681379Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.590s) 2022-03-17T21:59:02.3682371Z 2022-03-17T21:59:02.3683518Z ---------------------------------------------------------------------- 2022-03-17T21:59:02.3684303Z Ran 1 test in 0.590s 2022-03-17T21:59:02.3684680Z 2022-03-17T21:59:02.3684904Z OK (skipped=1) 2022-03-17T21:59:02.3685257Z 2022-03-17T21:59:02.3685544Z Generating XML reports... 2022-03-17T21:59:02.3713471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215901.xml 2022-03-17T21:59:04.9800988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:59:04.9812987Z 2022-03-17T21:59:04.9813946Z Running tests... 2022-03-17T21:59:04.9815159Z ---------------------------------------------------------------------- 2022-03-17T21:59:05.6119278Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103553 2022-03-17T21:59:05.6161167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103554 2022-03-17T21:59:05.6163755Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:59:06.7599561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:59:06.7600795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:59:06.7603055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:06.7604830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:06.7711748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:59:06.8613147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:59:08.9264962Z ok (3.945s) 2022-03-17T21:59:08.9265405Z 2022-03-17T21:59:08.9266970Z ---------------------------------------------------------------------- 2022-03-17T21:59:08.9267890Z Ran 1 test in 3.945s 2022-03-17T21:59:08.9268268Z 2022-03-17T21:59:08.9268461Z OK 2022-03-17T21:59:08.9268756Z 2022-03-17T21:59:08.9269025Z Generating XML reports... 2022-03-17T21:59:08.9341166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215904.xml 2022-03-17T21:59:11.6063586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:59:11.6070306Z 2022-03-17T21:59:11.6070549Z Running tests... 2022-03-17T21:59:11.6071708Z ---------------------------------------------------------------------- 2022-03-17T21:59:12.2072879Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103763 2022-03-17T21:59:12.2111867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103764 2022-03-17T21:59:12.2113203Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:59:13.3728691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:59:13.3731705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:59:13.3733672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:13.3831362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:13.3834911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:59:13.4756840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:59:13.8797167Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:59:13.8799551Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:59:13.8803128Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:59:13.8805417Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:59:23.0685533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:23.0695247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:23.0812093Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:59:23.0816394Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:59:23.1027071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:23.1028332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:23.1143093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:23.1144351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:25.7442856Z ok (14.137s) 2022-03-17T21:59:25.7445122Z 2022-03-17T21:59:25.7446306Z ---------------------------------------------------------------------- 2022-03-17T21:59:25.7447314Z Ran 1 test in 14.137s 2022-03-17T21:59:25.7447796Z 2022-03-17T21:59:25.7448029Z OK 2022-03-17T21:59:25.7448405Z 2022-03-17T21:59:25.7448697Z Generating XML reports... 2022-03-17T21:59:25.7522932Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215911.xml 2022-03-17T21:59:28.3387514Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:59:28.3399462Z 2022-03-17T21:59:28.3400434Z Running tests... 2022-03-17T21:59:28.3403280Z ---------------------------------------------------------------------- 2022-03-17T21:59:28.9774215Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103977 2022-03-17T21:59:28.9812787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103978 2022-03-17T21:59:28.9814097Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:59:30.1236880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:59:30.1237344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:59:30.1238358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:30.1239346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:30.1345373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:59:30.2253400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:59:30.6214981Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:59:30.6217670Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:59:30.6220954Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:59:30.6223138Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:59:39.7964662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:39.7972035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:42.5091731Z ok (14.168s) 2022-03-17T21:59:42.5092153Z 2022-03-17T21:59:42.5093477Z ---------------------------------------------------------------------- 2022-03-17T21:59:42.5094378Z Ran 1 test in 14.169s 2022-03-17T21:59:42.5094791Z 2022-03-17T21:59:42.5097361Z OK 2022-03-17T21:59:42.5097809Z 2022-03-17T21:59:42.5098092Z Generating XML reports... 2022-03-17T21:59:42.5174791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215928.xml 2022-03-17T21:59:45.1172459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:59:45.1184690Z 2022-03-17T21:59:45.1185215Z Running tests... 2022-03-17T21:59:45.1185848Z ---------------------------------------------------------------------- 2022-03-17T21:59:45.7201693Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104191 2022-03-17T21:59:45.7243010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104192 2022-03-17T21:59:45.7244347Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T21:59:46.7992733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:59:46.7994442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:59:46.7996357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:46.8094005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:59:46.8098833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:59:46.9009265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:59:56.4706475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:56.4713746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:56.5530501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:56.5531794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:59:59.2517452Z ok (14.133s) 2022-03-17T21:59:59.2517971Z 2022-03-17T21:59:59.2519087Z ---------------------------------------------------------------------- 2022-03-17T21:59:59.2520060Z Ran 1 test in 14.133s 2022-03-17T21:59:59.2520510Z 2022-03-17T21:59:59.2520723Z OK 2022-03-17T21:59:59.2521028Z 2022-03-17T21:59:59.2521338Z Generating XML reports... 2022-03-17T21:59:59.2599406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215945.xml 2022-03-17T22:00:01.8089326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:00:01.8100122Z 2022-03-17T22:00:01.8100651Z Running tests... 2022-03-17T22:00:01.8101343Z ---------------------------------------------------------------------- 2022-03-17T22:00:02.3999510Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... skip (0.590s) 2022-03-17T22:00:02.4000393Z 2022-03-17T22:00:02.4001451Z ---------------------------------------------------------------------- 2022-03-17T22:00:02.4002615Z Ran 1 test in 0.590s 2022-03-17T22:00:02.4003073Z 2022-03-17T22:00:02.4003334Z OK (skipped=1) 2022-03-17T22:00:02.4003680Z 2022-03-17T22:00:02.4003946Z Generating XML reports... 2022-03-17T22:00:02.4031083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220001.xml 2022-03-17T22:00:04.9980439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:00:04.9991188Z 2022-03-17T22:00:04.9991476Z Running tests... 2022-03-17T22:00:04.9992029Z ---------------------------------------------------------------------- 2022-03-17T22:00:05.6122647Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104472 2022-03-17T22:00:05.6165004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104473 2022-03-17T22:00:05.6166377Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:00:06.7688095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:00:06.7688781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:00:06.7689693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:06.7690511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:06.7794139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:00:06.8707896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:00:09.2267443Z ok (4.227s) 2022-03-17T22:00:09.2268050Z 2022-03-17T22:00:09.2268972Z ---------------------------------------------------------------------- 2022-03-17T22:00:09.2269727Z Ran 1 test in 4.227s 2022-03-17T22:00:09.2270088Z 2022-03-17T22:00:09.2270270Z OK 2022-03-17T22:00:09.2270560Z 2022-03-17T22:00:09.2270821Z Generating XML reports... 2022-03-17T22:00:09.2348299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220004.xml 2022-03-17T22:00:11.8089807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:00:11.8101934Z 2022-03-17T22:00:11.8102541Z Running tests... 2022-03-17T22:00:11.8103635Z ---------------------------------------------------------------------- 2022-03-17T22:00:12.4083600Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104682 2022-03-17T22:00:12.4118812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104683 2022-03-17T22:00:12.4120085Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:00:13.5829671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:00:13.5831986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:13.5833802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:00:13.5835601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:13.5937248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:00:13.6852989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:00:15.6220144Z ok (3.811s) 2022-03-17T22:00:15.6220679Z 2022-03-17T22:00:15.6221828Z ---------------------------------------------------------------------- 2022-03-17T22:00:15.6222814Z Ran 1 test in 3.812s 2022-03-17T22:00:15.6223274Z 2022-03-17T22:00:15.6223490Z OK 2022-03-17T22:00:15.6223801Z 2022-03-17T22:00:15.6224103Z Generating XML reports... 2022-03-17T22:00:15.6302162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220011.xml 2022-03-17T22:00:18.2099321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:00:18.2108407Z 2022-03-17T22:00:18.2108817Z Running tests... 2022-03-17T22:00:18.2110196Z ---------------------------------------------------------------------- 2022-03-17T22:00:18.8333854Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104892 2022-03-17T22:00:18.8374015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104893 2022-03-17T22:00:18.8375676Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:00:19.9820013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:00:19.9820585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:00:19.9821482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:19.9822276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:19.9928013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:00:20.0837141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:00:29.5224422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:29.5232073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:32.1704109Z ok (13.959s) 2022-03-17T22:00:32.1704628Z 2022-03-17T22:00:32.1705635Z ---------------------------------------------------------------------- 2022-03-17T22:00:32.1706422Z Ran 1 test in 13.959s 2022-03-17T22:00:32.1706810Z 2022-03-17T22:00:32.1706998Z OK 2022-03-17T22:00:32.1707744Z 2022-03-17T22:00:32.1708016Z Generating XML reports... 2022-03-17T22:00:32.1782873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220018.xml 2022-03-17T22:00:34.7213576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:00:34.7222877Z 2022-03-17T22:00:34.7223205Z Running tests... 2022-03-17T22:00:34.7224623Z ---------------------------------------------------------------------- 2022-03-17T22:00:35.3284212Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105106 2022-03-17T22:00:35.3317463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105107 2022-03-17T22:00:35.3318780Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:00:36.4412423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:00:36.4413340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:36.4413938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:00:36.4414692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:00:36.4424803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:00:36.5425562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:00:36.7913703Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T22:00:36.8936633Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T22:00:36.8938439Z Running test: batch_norm_net sync interval 2022-03-17T22:00:36.8939214Z 1 with iteration mapping 2022-03-17T22:00:36.8939831Z {0: 0, 1: 2} 2022-03-17T22:00:46.1785049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:46.1891414Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:46.1893710Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:46.1897331Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:46.1899436Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:46.3575738Z Running test: batch_norm_net sync interval 2022-03-17T22:00:46.3576597Z 1 with iteration mapping 2022-03-17T22:00:46.3577247Z {0: 0, 1: 3} 2022-03-17T22:00:46.3693479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:46.3812250Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:46.3814474Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:46.3817506Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:46.3819589Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:46.3954481Z Running test: batch_norm_net sync interval 2022-03-17T22:00:46.3955920Z 1 with iteration mapping 2022-03-17T22:00:46.3956569Z {0: 0, 1: 10} 2022-03-17T22:00:46.4054403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:46.4480355Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:46.4482731Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:46.4485802Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:46.4487916Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:46.4605095Z Running test: batch_norm_net sync interval 2022-03-17T22:00:46.4605912Z 1 with iteration mapping 2022-03-17T22:00:46.4606574Z {0: 5, 1: 7} 2022-03-17T22:00:55.6587119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:55.6592267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:55.6889464Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:55.6891717Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:55.6894717Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:55.6897373Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:55.7053606Z Running test: batch_norm_net sync interval 2022-03-17T22:00:55.7054537Z 1 with iteration mapping 2022-03-17T22:00:55.7055286Z {0: 5, 1: 8} 2022-03-17T22:00:55.7142781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:55.7144066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:55.7454562Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:55.7456730Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:55.7459732Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:55.7461807Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:55.7565603Z Running test: batch_norm_net sync interval 2022-03-17T22:00:55.7566506Z 1 with iteration mapping 2022-03-17T22:00:55.7567122Z {0: 5, 1: 15} 2022-03-17T22:00:55.7642663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:55.7643957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:55.8263183Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:55.8265592Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:55.8268664Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:55.8270821Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:55.8379571Z Running test: large_conv_model sync interval 2022-03-17T22:00:55.8380548Z 1 with iteration mapping 2022-03-17T22:00:55.8381312Z {0: 0, 1: 2} 2022-03-17T22:00:56.3405092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:56.5310683Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:56.5312816Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:56.5315725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:56.5318266Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:56.5640353Z Running test: large_conv_model sync interval 2022-03-17T22:00:56.5641144Z 1 with iteration mapping 2022-03-17T22:00:56.5641781Z {0: 0, 1: 3} 2022-03-17T22:00:56.7556561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:57.1431165Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:57.1433293Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:57.1436119Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:57.1438057Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:57.1694393Z Running test: large_conv_model sync interval 2022-03-17T22:00:57.1695471Z 1 with iteration mapping 2022-03-17T22:00:57.1696705Z {0: 0, 1: 10} 2022-03-17T22:00:57.3604288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:59.0288655Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:59.0290711Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:59.0293529Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:00:59.0295500Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:00:59.0507763Z Running test: large_conv_model sync interval 2022-03-17T22:00:59.0508700Z 1 with iteration mapping 2022-03-17T22:00:59.0509310Z {0: 5, 1: 7} 2022-03-17T22:00:59.5457948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:00:59.5459291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:00.6619816Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:00.6621886Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:00.6624759Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:00.6627426Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:00.6854594Z Running test: large_conv_model sync interval 2022-03-17T22:01:00.6854979Z 1 with iteration mapping 2022-03-17T22:01:00.6855264Z {0: 5, 1: 8} 2022-03-17T22:01:00.8758912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:00.8760371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:02.1919988Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:02.1922422Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:02.1925237Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:02.1927184Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:02.2124654Z Running test: large_conv_model sync interval 2022-03-17T22:01:02.2125984Z 1 with iteration mapping 2022-03-17T22:01:02.2126451Z {0: 5, 1: 15} 2022-03-17T22:01:02.4038132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:02.4039345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.0070091Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.0072261Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.0075186Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.0077235Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.0307089Z Running test: small_model sync interval 2022-03-17T22:01:05.0307468Z 1 with iteration mapping 2022-03-17T22:01:05.0307751Z {0: 0, 1: 2} 2022-03-17T22:01:05.0476409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.0778667Z Running test: small_model sync interval 2022-03-17T22:01:05.0779303Z 1 with iteration mapping 2022-03-17T22:01:05.0779755Z {0: 0, 1: 3} 2022-03-17T22:01:05.0905552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.1166226Z Running test: small_model sync interval 2022-03-17T22:01:05.1167174Z 1 with iteration mapping 2022-03-17T22:01:05.1167807Z {0: 0, 1: 10} 2022-03-17T22:01:05.1286883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.2066998Z Running test: small_model sync interval 2022-03-17T22:01:05.2067550Z 1 with iteration mapping 2022-03-17T22:01:05.2068013Z {0: 5, 1: 7} 2022-03-17T22:01:05.2180661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.2181576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.2671800Z Running test: small_model sync interval 2022-03-17T22:01:05.2672550Z 1 with iteration mapping 2022-03-17T22:01:05.2673029Z {0: 5, 1: 8} 2022-03-17T22:01:05.2788613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.2790015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.3337601Z Running test: small_model sync interval 2022-03-17T22:01:05.3338441Z 1 with iteration mapping 2022-03-17T22:01:05.3339097Z {0: 5, 1: 15} 2022-03-17T22:01:05.3457127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.3458464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.4438690Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.4439667Z 1 with iteration mapping 2022-03-17T22:01:05.4440329Z {0: 0, 1: 2} 2022-03-17T22:01:05.4570931Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.4573930Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.4576980Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.4579091Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.4610163Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.4611101Z 1 with iteration mapping 2022-03-17T22:01:05.4611771Z {0: 0, 1: 3} 2022-03-17T22:01:05.4740981Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.4743251Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.4746327Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.4748487Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.4778529Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.4780041Z 1 with iteration mapping 2022-03-17T22:01:05.4780733Z {0: 0, 1: 10} 2022-03-17T22:01:05.5081276Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.5083960Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.5086776Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.5088751Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.5118736Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.5119599Z 1 with iteration mapping 2022-03-17T22:01:05.5120239Z {0: 5, 1: 7} 2022-03-17T22:01:05.5162607Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:01:05.5365921Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.5368282Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.5371055Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.5372971Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.5407859Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.5408726Z 1 with iteration mapping 2022-03-17T22:01:05.5409345Z {0: 5, 1: 8} 2022-03-17T22:01:05.5660628Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.5662763Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.5665763Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.5667861Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.5711037Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.5711912Z 1 with iteration mapping 2022-03-17T22:01:05.5712550Z {0: 5, 1: 15} 2022-03-17T22:01:05.6121539Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6123850Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6126673Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6128592Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6161698Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.6162731Z 1 with iteration mapping 2022-03-17T22:01:05.6163362Z {0: 0, 1: 2} 2022-03-17T22:01:05.6207330Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:01:05.6281349Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6283965Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6286814Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6288719Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6321014Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.6321995Z 1 with iteration mapping 2022-03-17T22:01:05.6322625Z {0: 0, 1: 3} 2022-03-17T22:01:05.6461714Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6463724Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6466462Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6468414Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6498474Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.6499319Z 1 with iteration mapping 2022-03-17T22:01:05.6499940Z {0: 0, 1: 10} 2022-03-17T22:01:05.6808295Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6810334Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6813110Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.6815054Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.6849882Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.6850720Z 1 with iteration mapping 2022-03-17T22:01:05.6851354Z {0: 5, 1: 7} 2022-03-17T22:01:05.7073568Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.7075603Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.7078413Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.7080649Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.7113752Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.7114813Z 1 with iteration mapping 2022-03-17T22:01:05.7115569Z {0: 5, 1: 8} 2022-03-17T22:01:05.7358031Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.7360189Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.7363201Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.7365184Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.7400297Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:05.7401332Z 1 with iteration mapping 2022-03-17T22:01:05.7402225Z {0: 5, 1: 15} 2022-03-17T22:01:05.7865312Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.7867458Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.7870646Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:05.7872625Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:05.7917875Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:05.7918855Z 1 with iteration mapping 2022-03-17T22:01:05.7919625Z {0: 0, 1: 2} 2022-03-17T22:01:05.7972978Z registered hook 2022-03-17T22:01:05.7974603Z registered hook 2022-03-17T22:01:05.8059497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.8286778Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:05.8287689Z 1 with iteration mapping 2022-03-17T22:01:05.8288300Z {0: 0, 1: 3} 2022-03-17T22:01:05.8329515Z registered hook 2022-03-17T22:01:05.8330499Z registered hook 2022-03-17T22:01:05.8402187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.8682576Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:05.8683159Z 1 with iteration mapping 2022-03-17T22:01:05.8683589Z {0: 0, 1: 10} 2022-03-17T22:01:05.8725313Z registered hook 2022-03-17T22:01:05.8726301Z registered hook 2022-03-17T22:01:05.8806869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.9541224Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:05.9541870Z 1 with iteration mapping 2022-03-17T22:01:05.9542309Z {0: 5, 1: 7} 2022-03-17T22:01:05.9597773Z registered hook 2022-03-17T22:01:05.9600241Z registered hook 2022-03-17T22:01:05.9685387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:05.9686278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.0250210Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:06.0250801Z 1 with iteration mapping 2022-03-17T22:01:06.0251278Z {0: 5, 1: 8} 2022-03-17T22:01:06.0295508Z registered hook 2022-03-17T22:01:06.0296201Z registered hook 2022-03-17T22:01:06.0380023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.0380898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.1025046Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:06.1025629Z 1 with iteration mapping 2022-03-17T22:01:06.1026089Z {0: 5, 1: 15} 2022-03-17T22:01:06.1076053Z registered hook 2022-03-17T22:01:06.1076708Z registered hook 2022-03-17T22:01:06.1155526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.1156414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.2133676Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:06.2134563Z 1 with iteration mapping 2022-03-17T22:01:06.2135401Z {0: 0, 1: 2} 2022-03-17T22:01:06.2186408Z registered hook 2022-03-17T22:01:06.2189663Z registered hook 2022-03-17T22:01:06.2206033Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T22:01:06.2211765Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T22:01:06.2271095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.2276299Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T22:01:06.2278697Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T22:01:06.2882555Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:06.2883730Z 1 with iteration mapping 2022-03-17T22:01:06.2884415Z {0: 0, 1: 3} 2022-03-17T22:01:06.2921825Z registered hook 2022-03-17T22:01:06.2922643Z registered hook 2022-03-17T22:01:06.2976567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.3316325Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:06.3317325Z 1 with iteration mapping 2022-03-17T22:01:06.3318002Z {0: 0, 1: 10} 2022-03-17T22:01:06.3368651Z registered hook 2022-03-17T22:01:06.3369716Z registered hook 2022-03-17T22:01:06.3433462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.3947772Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:06.3949062Z 1 with iteration mapping 2022-03-17T22:01:06.3949732Z {0: 5, 1: 7} 2022-03-17T22:01:06.3989183Z registered hook 2022-03-17T22:01:06.3989865Z registered hook 2022-03-17T22:01:06.4039113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.4040424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.4387926Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:06.4388875Z 1 with iteration mapping 2022-03-17T22:01:06.4389519Z {0: 5, 1: 8} 2022-03-17T22:01:06.4433091Z registered hook 2022-03-17T22:01:06.4433780Z registered hook 2022-03-17T22:01:06.4491413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.4492369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.4894328Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:06.4894940Z 1 with iteration mapping 2022-03-17T22:01:06.4895378Z {0: 5, 1: 15} 2022-03-17T22:01:06.4943857Z registered hook 2022-03-17T22:01:06.4944517Z registered hook 2022-03-17T22:01:06.5005163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.5006468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.5718577Z Running test: batch_norm_net sync interval 2022-03-17T22:01:06.5719302Z 2 with iteration mapping 2022-03-17T22:01:06.5719636Z {0: 0, 1: 2} 2022-03-17T22:01:06.5833560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.5901505Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.5903256Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.5905740Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.5908438Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.6031659Z Running test: batch_norm_net sync interval 2022-03-17T22:01:06.6032464Z 2 with iteration mapping 2022-03-17T22:01:06.6033080Z {0: 0, 1: 3} 2022-03-17T22:01:06.6128881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.6218311Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.6220645Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.6223506Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.6225435Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.6351896Z Running test: batch_norm_net sync interval 2022-03-17T22:01:06.6352671Z 2 with iteration mapping 2022-03-17T22:01:06.6353300Z {0: 0, 1: 10} 2022-03-17T22:01:06.6447855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.6770686Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.6772730Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.6775597Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.6777540Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.6910062Z Running test: batch_norm_net sync interval 2022-03-17T22:01:06.6910830Z 2 with iteration mapping 2022-03-17T22:01:06.6911479Z {0: 5, 1: 7} 2022-03-17T22:01:06.7010241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.7239520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.7243037Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.7245050Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.7247907Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.7249874Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.7364800Z Running test: batch_norm_net sync interval 2022-03-17T22:01:06.7365567Z 2 with iteration mapping 2022-03-17T22:01:06.7366217Z {0: 5, 1: 8} 2022-03-17T22:01:06.7440832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.7442329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.7693701Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.7696115Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.7699014Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.7700974Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.7815069Z Running test: batch_norm_net sync interval 2022-03-17T22:01:06.7815833Z 2 with iteration mapping 2022-03-17T22:01:06.7816490Z {0: 5, 1: 15} 2022-03-17T22:01:06.7904013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.7905274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:06.8364942Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.8367025Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.8369845Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:06.8371800Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:06.8495094Z Running test: large_conv_model sync interval 2022-03-17T22:01:06.8495866Z 3 with iteration mapping 2022-03-17T22:01:06.8496508Z {0: 0, 1: 2} 2022-03-17T22:01:07.0501241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:07.2365060Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:07.2367139Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:07.2370064Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:07.2372062Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:07.2554063Z Running test: large_conv_model sync interval 2022-03-17T22:01:07.2554904Z 3 with iteration mapping 2022-03-17T22:01:07.2555566Z {0: 0, 1: 3} 2022-03-17T22:01:07.4475147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:07.8160240Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:07.8163049Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:07.8165958Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:07.8168032Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:07.8365960Z Running test: large_conv_model sync interval 2022-03-17T22:01:07.8366547Z 3 with iteration mapping 2022-03-17T22:01:07.8366997Z {0: 0, 1: 10} 2022-03-17T22:01:08.0283314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:09.6771871Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:09.6773395Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:09.6775952Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:09.6778066Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:09.6965478Z Running test: large_conv_model sync interval 2022-03-17T22:01:09.6966289Z 3 with iteration mapping 2022-03-17T22:01:09.6966926Z {0: 5, 1: 7} 2022-03-17T22:01:09.8876582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:09.8877879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:10.9984661Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:10.9987111Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:10.9990072Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:10.9992155Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:11.0130529Z Running test: large_conv_model sync interval 2022-03-17T22:01:11.0131436Z 3 with iteration mapping 2022-03-17T22:01:11.2051619Z {0: 5, 1: 8} 2022-03-17T22:01:11.2052755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:11.2054141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:12.5024446Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:12.5026807Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:12.5029596Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:12.5031566Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:12.5224688Z Running test: large_conv_model sync interval 2022-03-17T22:01:12.5225462Z 3 with iteration mapping 2022-03-17T22:01:12.5226085Z {0: 5, 1: 15} 2022-03-17T22:01:12.7147110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:12.7148323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.3004685Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.3007010Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.3010157Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.3012355Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.3200479Z Running test: small_model sync interval 2022-03-17T22:01:15.3201251Z 4 with iteration mapping 2022-03-17T22:01:15.3202060Z {0: 0, 1: 2} 2022-03-17T22:01:15.3336848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.3573772Z Running test: small_model sync interval 2022-03-17T22:01:15.3574317Z 4 with iteration mapping 2022-03-17T22:01:15.3575068Z {0: 0, 1: 3} 2022-03-17T22:01:15.3703820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.3869441Z Running test: small_model sync interval 2022-03-17T22:01:15.3870058Z 4 with iteration mapping 2022-03-17T22:01:15.3870544Z {0: 0, 1: 10} 2022-03-17T22:01:15.3996503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.4352816Z Running test: small_model sync interval 2022-03-17T22:01:15.4353590Z 4 with iteration mapping 2022-03-17T22:01:15.4354223Z {0: 5, 1: 7} 2022-03-17T22:01:15.4476353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.4477215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.4805253Z Running test: small_model sync interval 2022-03-17T22:01:15.4805799Z 4 with iteration mapping 2022-03-17T22:01:15.4806254Z {0: 5, 1: 8} 2022-03-17T22:01:15.4931538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.4932374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.5209822Z Running test: small_model sync interval 2022-03-17T22:01:15.5210583Z 4 with iteration mapping 2022-03-17T22:01:15.5211722Z {0: 5, 1: 15} 2022-03-17T22:01:15.5329135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.5330437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.5785060Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.5785972Z 5 with iteration mapping 2022-03-17T22:01:15.5786600Z {0: 0, 1: 2} 2022-03-17T22:01:15.5893590Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.5895706Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.5898575Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.5900558Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.5933295Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.5933915Z 5 with iteration mapping 2022-03-17T22:01:15.5934372Z {0: 0, 1: 3} 2022-03-17T22:01:15.6031887Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6033370Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6035370Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6036998Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6068771Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.6069646Z 5 with iteration mapping 2022-03-17T22:01:15.6070279Z {0: 0, 1: 10} 2022-03-17T22:01:15.6269678Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6271711Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6274517Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6276440Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6305363Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.6306232Z 5 with iteration mapping 2022-03-17T22:01:15.6307201Z {0: 5, 1: 7} 2022-03-17T22:01:15.6475518Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6477613Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6480496Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6482694Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6516412Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.6517479Z 5 with iteration mapping 2022-03-17T22:01:15.6518267Z {0: 5, 1: 8} 2022-03-17T22:01:15.6702164Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6704385Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6707298Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.6709299Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.6741444Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.6742378Z 5 with iteration mapping 2022-03-17T22:01:15.6743019Z {0: 5, 1: 15} 2022-03-17T22:01:15.7018866Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7020922Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7023778Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7025761Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7079697Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.7080547Z 6 with iteration mapping 2022-03-17T22:01:15.7081201Z {0: 0, 1: 2} 2022-03-17T22:01:15.7175015Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7177219Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7180400Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7182409Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7214824Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.7215864Z 6 with iteration mapping 2022-03-17T22:01:15.7216628Z {0: 0, 1: 3} 2022-03-17T22:01:15.7330239Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7332430Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7335381Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7337334Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7371264Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.7372269Z 6 with iteration mapping 2022-03-17T22:01:15.7373080Z {0: 0, 1: 10} 2022-03-17T22:01:15.7584056Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7586095Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7589308Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7591282Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7624826Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.7625672Z 6 with iteration mapping 2022-03-17T22:01:15.7626335Z {0: 5, 1: 7} 2022-03-17T22:01:15.7798435Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7800471Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7803468Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.7805420Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.7836951Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.7837796Z 6 with iteration mapping 2022-03-17T22:01:15.7838409Z {0: 5, 1: 8} 2022-03-17T22:01:15.8023442Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.8025479Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.8028318Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.8030265Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.8068507Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:15.8069373Z 6 with iteration mapping 2022-03-17T22:01:15.8069994Z {0: 5, 1: 15} 2022-03-17T22:01:15.8374761Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.8376782Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.8379580Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:15.8381536Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:15.8412917Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:15.8413836Z 7 with iteration mapping 2022-03-17T22:01:15.8414853Z {0: 0, 1: 2} 2022-03-17T22:01:15.8540802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.8711741Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:15.8712564Z 7 with iteration mapping 2022-03-17T22:01:15.8713187Z {0: 0, 1: 3} 2022-03-17T22:01:15.8835777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.9010108Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:15.9010919Z 7 with iteration mapping 2022-03-17T22:01:15.9011541Z {0: 0, 1: 10} 2022-03-17T22:01:15.9139873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.9472172Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:15.9473104Z 7 with iteration mapping 2022-03-17T22:01:15.9473755Z {0: 5, 1: 7} 2022-03-17T22:01:15.9610564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.9611856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.9873846Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:15.9874729Z 7 with iteration mapping 2022-03-17T22:01:15.9875339Z {0: 5, 1: 8} 2022-03-17T22:01:15.9980436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:15.9981180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.0265805Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:16.0266776Z 7 with iteration mapping 2022-03-17T22:01:16.0267406Z {0: 5, 1: 15} 2022-03-17T22:01:16.0378793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.0380021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.0768024Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:16.0768838Z 8 with iteration mapping 2022-03-17T22:01:16.0769473Z {0: 0, 1: 2} 2022-03-17T22:01:16.0889734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.1103075Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:16.1103894Z 8 with iteration mapping 2022-03-17T22:01:16.1104525Z {0: 0, 1: 3} 2022-03-17T22:01:16.1204825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.1360115Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:16.1360978Z 8 with iteration mapping 2022-03-17T22:01:16.1361621Z {0: 0, 1: 10} 2022-03-17T22:01:16.1471118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.1811423Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:16.1811832Z 8 with iteration mapping 2022-03-17T22:01:16.1812122Z {0: 5, 1: 7} 2022-03-17T22:01:16.1929081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.1929657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.2143798Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:16.2144202Z 8 with iteration mapping 2022-03-17T22:01:16.2144488Z {0: 5, 1: 8} 2022-03-17T22:01:16.2249951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.2250521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.2493352Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:16.2494230Z 8 with iteration mapping 2022-03-17T22:01:16.2494859Z {0: 5, 1: 15} 2022-03-17T22:01:16.2608814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.2610099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.2975663Z Running test: batch_norm_net sync interval 2022-03-17T22:01:16.2976217Z 1 with iteration mapping 2022-03-17T22:01:16.2976641Z {0: 0, 1: 2} 2022-03-17T22:01:16.3047382Z Running test: batch_norm_net sync interval 2022-03-17T22:01:16.3047959Z 1 with iteration mapping 2022-03-17T22:01:16.3048401Z {0: 0, 1: 3} 2022-03-17T22:01:16.3088622Z Running test: batch_norm_net sync interval 2022-03-17T22:01:16.3089196Z 1 with iteration mapping 2022-03-17T22:01:16.3089670Z {0: 0, 1: 10} 2022-03-17T22:01:16.3143480Z Running test: batch_norm_net sync interval 2022-03-17T22:01:16.3144269Z 1 with iteration mapping 2022-03-17T22:01:16.3144899Z {0: 5, 1: 7} 2022-03-17T22:01:16.3253778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.3255630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.3446927Z Running test: batch_norm_net sync interval 2022-03-17T22:01:16.3447728Z 1 with iteration mapping 2022-03-17T22:01:16.3448366Z {0: 5, 1: 8} 2022-03-17T22:01:16.3545068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.3546378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.3744451Z Running test: batch_norm_net sync interval 2022-03-17T22:01:16.3745227Z 1 with iteration mapping 2022-03-17T22:01:16.3745862Z {0: 5, 1: 15} 2022-03-17T22:01:16.3847516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.3848824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.4042472Z Running test: large_conv_model sync interval 2022-03-17T22:01:16.4043288Z 1 with iteration mapping 2022-03-17T22:01:16.4043917Z {0: 0, 1: 2} 2022-03-17T22:01:16.4099512Z Running test: large_conv_model sync interval 2022-03-17T22:01:16.4100301Z 1 with iteration mapping 2022-03-17T22:01:16.4100946Z {0: 0, 1: 3} 2022-03-17T22:01:16.4154453Z Running test: large_conv_model sync interval 2022-03-17T22:01:16.4155288Z 1 with iteration mapping 2022-03-17T22:01:16.4155931Z {0: 0, 1: 10} 2022-03-17T22:01:16.4215211Z Running test: large_conv_model sync interval 2022-03-17T22:01:16.4216095Z 1 with iteration mapping 2022-03-17T22:01:16.4216715Z {0: 5, 1: 7} 2022-03-17T22:01:16.6140988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:16.6142255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:17.3571843Z Running test: large_conv_model sync interval 2022-03-17T22:01:17.3572800Z 1 with iteration mapping 2022-03-17T22:01:17.3573449Z {0: 5, 1: 8} 2022-03-17T22:01:17.5636337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:17.5638007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:18.3185990Z Running test: large_conv_model sync interval 2022-03-17T22:01:18.3186765Z 1 with iteration mapping 2022-03-17T22:01:18.3187424Z {0: 5, 1: 15} 2022-03-17T22:01:18.5123836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:18.5125061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.2577930Z Running test: small_model sync interval 2022-03-17T22:01:19.2578685Z 1 with iteration mapping 2022-03-17T22:01:19.2579327Z {0: 0, 1: 2} 2022-03-17T22:01:19.2651073Z Running test: small_model sync interval 2022-03-17T22:01:19.2651878Z 1 with iteration mapping 2022-03-17T22:01:19.2652520Z {0: 0, 1: 3} 2022-03-17T22:01:19.2720204Z Running test: small_model sync interval 2022-03-17T22:01:19.2720990Z 1 with iteration mapping 2022-03-17T22:01:19.2784027Z {0: 0, 1: 10} 2022-03-17T22:01:19.2784830Z Running test: small_model sync interval 2022-03-17T22:01:19.2785646Z 1 with iteration mapping 2022-03-17T22:01:19.2786314Z {0: 5, 1: 7} 2022-03-17T22:01:19.2932483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.2946362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.3207586Z Running test: small_model sync interval 2022-03-17T22:01:19.3208366Z 1 with iteration mapping 2022-03-17T22:01:19.3208989Z {0: 5, 1: 8} 2022-03-17T22:01:19.3342863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.3344196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.3604344Z Running test: small_model sync interval 2022-03-17T22:01:19.3605218Z 1 with iteration mapping 2022-03-17T22:01:19.3605961Z {0: 5, 1: 15} 2022-03-17T22:01:19.3737415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.3738657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.3998626Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.3999519Z 1 with iteration mapping 2022-03-17T22:01:19.4000162Z {0: 0, 1: 2} 2022-03-17T22:01:19.4048315Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4049224Z 1 with iteration mapping 2022-03-17T22:01:19.4049878Z {0: 0, 1: 3} 2022-03-17T22:01:19.4088878Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4089787Z 1 with iteration mapping 2022-03-17T22:01:19.4090449Z {0: 0, 1: 10} 2022-03-17T22:01:19.4135452Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4136442Z 1 with iteration mapping 2022-03-17T22:01:19.4137069Z {0: 5, 1: 7} 2022-03-17T22:01:19.4313896Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4314796Z 1 with iteration mapping 2022-03-17T22:01:19.4315439Z {0: 5, 1: 8} 2022-03-17T22:01:19.4489103Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4489982Z 1 with iteration mapping 2022-03-17T22:01:19.4490599Z {0: 5, 1: 15} 2022-03-17T22:01:19.4665977Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4666828Z 1 with iteration mapping 2022-03-17T22:01:19.4667739Z {0: 0, 1: 2} 2022-03-17T22:01:19.4706332Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4707275Z 1 with iteration mapping 2022-03-17T22:01:19.4707953Z {0: 0, 1: 3} 2022-03-17T22:01:19.4750910Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4751828Z 1 with iteration mapping 2022-03-17T22:01:19.4752554Z {0: 0, 1: 10} 2022-03-17T22:01:19.4791783Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4792618Z 1 with iteration mapping 2022-03-17T22:01:19.4793257Z {0: 5, 1: 7} 2022-03-17T22:01:19.4961019Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.4962104Z 1 with iteration mapping 2022-03-17T22:01:19.4962752Z {0: 5, 1: 8} 2022-03-17T22:01:19.5135037Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T22:01:19.5136166Z 1 with iteration mapping 2022-03-17T22:01:19.5136904Z {0: 5, 1: 15} 2022-03-17T22:01:19.5307116Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:19.5307930Z 1 with iteration mapping 2022-03-17T22:01:19.5308537Z {0: 0, 1: 2} 2022-03-17T22:01:19.5373542Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:19.5374663Z 1 with iteration mapping 2022-03-17T22:01:19.5375301Z {0: 0, 1: 3} 2022-03-17T22:01:19.5438127Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:19.5438949Z 1 with iteration mapping 2022-03-17T22:01:19.5439579Z {0: 0, 1: 10} 2022-03-17T22:01:19.5503546Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:19.5504542Z 1 with iteration mapping 2022-03-17T22:01:19.5505224Z {0: 5, 1: 7} 2022-03-17T22:01:19.5634398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.5635882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.5874040Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:19.5875056Z 1 with iteration mapping 2022-03-17T22:01:19.5875860Z {0: 5, 1: 8} 2022-03-17T22:01:19.6006715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.6008256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.6264446Z Running test: small_model_allreduce_hook sync interval 2022-03-17T22:01:19.6265431Z 1 with iteration mapping 2022-03-17T22:01:19.6266248Z {0: 5, 1: 15} 2022-03-17T22:01:19.6397028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.6398539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.6645774Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:19.6646649Z 1 with iteration mapping 2022-03-17T22:01:19.6647298Z {0: 0, 1: 2} 2022-03-17T22:01:19.6714252Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:19.6715276Z 1 with iteration mapping 2022-03-17T22:01:19.6716081Z {0: 0, 1: 3} 2022-03-17T22:01:19.6780680Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:19.6781519Z 1 with iteration mapping 2022-03-17T22:01:19.6782144Z {0: 0, 1: 10} 2022-03-17T22:01:19.6847543Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:19.6848389Z 1 with iteration mapping 2022-03-17T22:01:19.6849026Z {0: 5, 1: 7} 2022-03-17T22:01:19.6972369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.6973597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.7222368Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:19.7223204Z 1 with iteration mapping 2022-03-17T22:01:19.7223844Z {0: 5, 1: 8} 2022-03-17T22:01:19.7357327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.7358678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.7610715Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T22:01:19.7611572Z 1 with iteration mapping 2022-03-17T22:01:19.7612225Z {0: 5, 1: 15} 2022-03-17T22:01:19.7741298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:19.7742588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:22.9168263Z ok (48.194s) 2022-03-17T22:01:22.9168651Z 2022-03-17T22:01:22.9169729Z ---------------------------------------------------------------------- 2022-03-17T22:01:22.9170514Z Ran 1 test in 48.194s 2022-03-17T22:01:22.9171287Z 2022-03-17T22:01:22.9171478Z OK 2022-03-17T22:01:22.9171774Z 2022-03-17T22:01:22.9172057Z Generating XML reports... 2022-03-17T22:01:22.9248800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220034.xml 2022-03-17T22:01:25.5062815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:01:25.5073994Z 2022-03-17T22:01:25.5074709Z Running tests... 2022-03-17T22:01:25.5076255Z ---------------------------------------------------------------------- 2022-03-17T22:01:26.1079216Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105452 2022-03-17T22:01:26.1113629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105453 2022-03-17T22:01:26.1114973Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:01:27.2752049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:01:27.2753124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:01:27.2753725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:01:27.2754488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:01:27.2766315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:01:27.3766852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:01:36.8619437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:36.8621059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:36.8956544Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:36.8959184Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:36.8962300Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:36.8964306Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:36.9844455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:36.9845694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:01:36.9997835Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T22:01:36.9999923Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T22:01:39.5363345Z ok (14.029s) 2022-03-17T22:01:39.5363843Z 2022-03-17T22:01:39.5365028Z ---------------------------------------------------------------------- 2022-03-17T22:01:39.5365991Z Ran 1 test in 14.029s 2022-03-17T22:01:39.5366431Z 2022-03-17T22:01:39.5366665Z OK 2022-03-17T22:01:39.5367027Z 2022-03-17T22:01:39.5367817Z Generating XML reports... 2022-03-17T22:01:39.5444889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220125.xml 2022-03-17T22:01:42.1103514Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:01:42.1115028Z 2022-03-17T22:01:42.1115411Z Running tests... 2022-03-17T22:01:42.1115952Z ---------------------------------------------------------------------- 2022-03-17T22:01:42.7115570Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105666 2022-03-17T22:01:42.7148600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105667 2022-03-17T22:01:42.7149965Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:01:43.8705115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:01:43.8717538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:01:43.8718056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:01:43.8718707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:01:43.8719197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:01:43.9720607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:01:56.2456543Z ok (14.134s) 2022-03-17T22:01:56.2457339Z 2022-03-17T22:01:56.2458456Z ---------------------------------------------------------------------- 2022-03-17T22:01:56.2459242Z Ran 1 test in 14.134s 2022-03-17T22:01:56.2459586Z 2022-03-17T22:01:56.2459827Z OK 2022-03-17T22:01:56.2460126Z 2022-03-17T22:01:56.2460403Z Generating XML reports... 2022-03-17T22:01:56.2536560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220142.xml 2022-03-17T22:01:58.7850734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:01:58.7861392Z 2022-03-17T22:01:58.7862159Z Running tests... 2022-03-17T22:01:58.7863821Z ---------------------------------------------------------------------- 2022-03-17T22:01:59.3851933Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105880 2022-03-17T22:01:59.3883079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105881 2022-03-17T22:01:59.3884687Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:00.5423207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:00.5424192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:00.5424783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:00.5425555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:00.5437835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:02:00.6433893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:00.6639924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:02:00.6641307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:02:00.6643905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:02:00.6743103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:02:02.5968043Z ok (3.810s) 2022-03-17T22:02:02.5968438Z 2022-03-17T22:02:02.5969415Z ---------------------------------------------------------------------- 2022-03-17T22:02:02.5970235Z Ran 1 test in 3.811s 2022-03-17T22:02:02.5970645Z 2022-03-17T22:02:02.5970828Z OK 2022-03-17T22:02:02.5971127Z 2022-03-17T22:02:02.5971410Z Generating XML reports... 2022-03-17T22:02:02.6052243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220158.xml 2022-03-17T22:02:05.1860213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:02:05.1873623Z 2022-03-17T22:02:05.1873924Z Running tests... 2022-03-17T22:02:05.1874456Z ---------------------------------------------------------------------- 2022-03-17T22:02:05.8048641Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106096 2022-03-17T22:02:05.8089271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106097 2022-03-17T22:02:05.8090600Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:06.9469441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:06.9470802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:06.9472659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:06.9474441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:06.9584029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:02:07.0488602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:07.0697693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:02:07.0699633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:02:07.0701609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:02:07.0703388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:02:08.8188468Z ok (3.631s) 2022-03-17T22:02:08.8189116Z 2022-03-17T22:02:08.8190226Z ---------------------------------------------------------------------- 2022-03-17T22:02:08.8191211Z Ran 1 test in 3.631s 2022-03-17T22:02:08.8191647Z 2022-03-17T22:02:08.8191877Z OK 2022-03-17T22:02:08.8192298Z 2022-03-17T22:02:08.8192731Z Generating XML reports... 2022-03-17T22:02:08.8265339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220205.xml 2022-03-17T22:02:11.5151666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:02:11.5162046Z 2022-03-17T22:02:11.5162340Z Running tests... 2022-03-17T22:02:11.5163014Z ---------------------------------------------------------------------- 2022-03-17T22:02:12.1514377Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106312 2022-03-17T22:02:12.1553168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106313 2022-03-17T22:02:12.1554478Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:13.2822013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:13.2822677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:13.2823813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:13.2926696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:13.2930028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:02:13.3937084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:22.9741168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:02:22.9744788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:02:22.9881632Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:02:22.9886200Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:02:25.6884342Z ok (14.171s) 2022-03-17T22:02:25.6884930Z 2022-03-17T22:02:25.6886048Z ---------------------------------------------------------------------- 2022-03-17T22:02:25.6886990Z Ran 1 test in 14.172s 2022-03-17T22:02:25.6887433Z 2022-03-17T22:02:25.6887706Z OK 2022-03-17T22:02:25.6888661Z 2022-03-17T22:02:25.6888984Z Generating XML reports... 2022-03-17T22:02:25.6968015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220211.xml 2022-03-17T22:02:28.2662192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:02:28.2673045Z 2022-03-17T22:02:28.2673449Z Running tests... 2022-03-17T22:02:28.2674614Z ---------------------------------------------------------------------- 2022-03-17T22:02:28.8644506Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106526 2022-03-17T22:02:28.8683691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106527 2022-03-17T22:02:28.8685312Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:29.9847776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:29.9848936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:29.9849534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:29.9850297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:29.9956622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:02:30.0869140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:39.3918167Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:02:39.6682736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:02:39.6684214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:02:42.2988509Z ok (14.031s) 2022-03-17T22:02:42.2989106Z 2022-03-17T22:02:42.2990269Z ---------------------------------------------------------------------- 2022-03-17T22:02:42.2991216Z Ran 1 test in 14.032s 2022-03-17T22:02:42.2991706Z 2022-03-17T22:02:42.2991916Z OK 2022-03-17T22:02:42.2992229Z 2022-03-17T22:02:42.2992505Z Generating XML reports... 2022-03-17T22:02:42.3066594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220228.xml 2022-03-17T22:02:44.8696709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:02:44.8710356Z 2022-03-17T22:02:44.8710683Z Running tests... 2022-03-17T22:02:44.8711332Z ---------------------------------------------------------------------- 2022-03-17T22:02:45.5050924Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106740 2022-03-17T22:02:45.5092118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106741 2022-03-17T22:02:45.5093701Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:46.6849208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:46.6851523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:46.6854500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:46.6856862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:46.6965723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:02:46.7872101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:48.5179116Z ok (3.646s) 2022-03-17T22:02:48.5179702Z 2022-03-17T22:02:48.5180964Z ---------------------------------------------------------------------- 2022-03-17T22:02:48.5181983Z Ran 1 test in 3.647s 2022-03-17T22:02:48.5182407Z 2022-03-17T22:02:48.5182592Z OK 2022-03-17T22:02:48.5182902Z 2022-03-17T22:02:48.5183210Z Generating XML reports... 2022-03-17T22:02:48.5262020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220244.xml 2022-03-17T22:02:51.0986992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:02:51.0997978Z 2022-03-17T22:02:51.0998288Z Running tests... 2022-03-17T22:02:51.0998960Z ---------------------------------------------------------------------- 2022-03-17T22:02:51.6991450Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106950 2022-03-17T22:02:51.7024379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106951 2022-03-17T22:02:51.7025964Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:52.8339769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:52.8340891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:52.8341500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:52.8342272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:52.8449757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:02:52.9351981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:54.7120926Z ok (3.612s) 2022-03-17T22:02:54.7121423Z 2022-03-17T22:02:54.7122668Z ---------------------------------------------------------------------- 2022-03-17T22:02:54.7123655Z Ran 1 test in 3.612s 2022-03-17T22:02:54.7124107Z 2022-03-17T22:02:54.7124355Z OK 2022-03-17T22:02:54.7124847Z 2022-03-17T22:02:54.7125180Z Generating XML reports... 2022-03-17T22:02:54.7205857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220251.xml 2022-03-17T22:02:57.3234468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:02:57.3248227Z 2022-03-17T22:02:57.3248587Z Running tests... 2022-03-17T22:02:57.3249236Z ---------------------------------------------------------------------- 2022-03-17T22:02:57.9446332Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107166 2022-03-17T22:02:57.9486098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107167 2022-03-17T22:02:57.9487501Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:02:59.0686739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:02:59.0689109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:02:59.0691932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:59.0694664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:02:59.0701366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:02:59.0702697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:00.9573981Z ok (3.632s) 2022-03-17T22:03:00.9574515Z 2022-03-17T22:03:00.9575757Z ---------------------------------------------------------------------- 2022-03-17T22:03:00.9576523Z Ran 1 test in 3.633s 2022-03-17T22:03:00.9576952Z 2022-03-17T22:03:00.9577280Z OK 2022-03-17T22:03:00.9577659Z 2022-03-17T22:03:00.9577956Z Generating XML reports... 2022-03-17T22:03:00.9655542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220257.xml 2022-03-17T22:03:03.6428536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:03.6442438Z 2022-03-17T22:03:03.6442791Z Running tests... 2022-03-17T22:03:03.6443957Z ---------------------------------------------------------------------- 2022-03-17T22:03:03.6460553Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:03:03.6461584Z 2022-03-17T22:03:03.6462409Z ---------------------------------------------------------------------- 2022-03-17T22:03:03.6463185Z Ran 1 test in 0.002s 2022-03-17T22:03:03.6463535Z 2022-03-17T22:03:03.6463769Z OK (skipped=1) 2022-03-17T22:03:03.6464120Z 2022-03-17T22:03:03.6464397Z Generating XML reports... 2022-03-17T22:03:03.6502224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220303.xml 2022-03-17T22:03:06.1560043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:06.1574934Z 2022-03-17T22:03:06.1575092Z Running tests... 2022-03-17T22:03:06.1577068Z ---------------------------------------------------------------------- 2022-03-17T22:03:06.7946408Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107443 2022-03-17T22:03:06.7981648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107444 2022-03-17T22:03:06.7982989Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:07.9449149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:07.9450398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:07.9452254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:07.9454016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:07.9556437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:08.0462309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:08.0670404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:03:08.0671896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:03:08.0674239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:08.0774829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:09.9080660Z ok (3.750s) 2022-03-17T22:03:09.9081096Z 2022-03-17T22:03:09.9082366Z ---------------------------------------------------------------------- 2022-03-17T22:03:09.9083326Z Ran 1 test in 3.750s 2022-03-17T22:03:09.9083771Z 2022-03-17T22:03:09.9084005Z OK 2022-03-17T22:03:09.9084386Z 2022-03-17T22:03:09.9084824Z Generating XML reports... 2022-03-17T22:03:09.9164614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220306.xml 2022-03-17T22:03:12.4984103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:12.4994333Z 2022-03-17T22:03:12.4994673Z Running tests... 2022-03-17T22:03:12.4995215Z ---------------------------------------------------------------------- 2022-03-17T22:03:13.0960958Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107665 2022-03-17T22:03:13.1001646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107666 2022-03-17T22:03:13.1003173Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:14.2472143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:14.2473420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:14.2475716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:14.2573123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:14.2594252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:14.3585716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:16.0096860Z skip (3.509s) 2022-03-17T22:03:16.0097299Z 2022-03-17T22:03:16.0098259Z ---------------------------------------------------------------------- 2022-03-17T22:03:16.0099140Z Ran 1 test in 3.510s 2022-03-17T22:03:16.0099501Z 2022-03-17T22:03:16.0099732Z OK (skipped=1) 2022-03-17T22:03:16.0100080Z 2022-03-17T22:03:16.0100347Z Generating XML reports... 2022-03-17T22:03:16.0178011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220312.xml 2022-03-17T22:03:18.6559044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:18.6571379Z 2022-03-17T22:03:18.6571638Z Running tests... 2022-03-17T22:03:19.2867611Z ---------------------------------------------------------------------- 2022-03-17T22:03:19.2868992Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107875 2022-03-17T22:03:19.2904718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107876 2022-03-17T22:03:19.2906031Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:20.4143053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:20.4143944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:20.4144560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:20.4145317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:20.4253001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:20.5157915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:22.2992264Z ok (3.642s) 2022-03-17T22:03:22.2992654Z 2022-03-17T22:03:22.2993658Z ---------------------------------------------------------------------- 2022-03-17T22:03:22.2994794Z Ran 1 test in 3.642s 2022-03-17T22:03:22.2995247Z 2022-03-17T22:03:22.2995490Z OK 2022-03-17T22:03:22.2995828Z 2022-03-17T22:03:22.2996170Z Generating XML reports... 2022-03-17T22:03:22.3069936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220318.xml 2022-03-17T22:03:25.0142780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:25.0153298Z 2022-03-17T22:03:25.0153918Z Running tests... 2022-03-17T22:03:25.0155366Z ---------------------------------------------------------------------- 2022-03-17T22:03:25.6368669Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108085 2022-03-17T22:03:25.6408416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108086 2022-03-17T22:03:25.6410000Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:26.7742967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:26.7744533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:26.7747418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:26.7844161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:26.7864589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:26.8857745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:26.9096231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:03:26.9197342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:03:26.9199484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:26.9201300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:26.9473499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T22:03:26.9474712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T22:03:26.9476600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T22:03:26.9478394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T22:03:26.9615115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T22:03:26.9616504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T22:03:26.9618309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T22:03:26.9620067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T22:03:28.8495942Z ok (3.834s) 2022-03-17T22:03:28.8496365Z 2022-03-17T22:03:28.8497426Z ---------------------------------------------------------------------- 2022-03-17T22:03:28.8498191Z Ran 1 test in 3.834s 2022-03-17T22:03:28.8498575Z 2022-03-17T22:03:28.8499386Z OK 2022-03-17T22:03:28.8499698Z 2022-03-17T22:03:28.8499989Z Generating XML reports... 2022-03-17T22:03:28.8577663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220325.xml 2022-03-17T22:03:31.5465263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:31.5475711Z 2022-03-17T22:03:31.5476324Z Running tests... 2022-03-17T22:03:31.5477554Z ---------------------------------------------------------------------- 2022-03-17T22:03:32.1601142Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108319 2022-03-17T22:03:32.1636660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108320 2022-03-17T22:03:32.1638211Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:33.3079180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:33.3080514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:33.3083446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:33.3180957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:33.3183502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:33.4106014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:33.4318508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:03:33.4319995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:03:33.4322204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:33.4422652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:35.2739494Z ok (3.726s) 2022-03-17T22:03:35.2740088Z 2022-03-17T22:03:35.2741228Z ---------------------------------------------------------------------- 2022-03-17T22:03:35.2742019Z Ran 1 test in 3.726s 2022-03-17T22:03:35.2742432Z 2022-03-17T22:03:35.2742621Z OK 2022-03-17T22:03:35.2742912Z 2022-03-17T22:03:35.2743178Z Generating XML reports... 2022-03-17T22:03:35.2820365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220331.xml 2022-03-17T22:03:37.9066376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:37.9077058Z 2022-03-17T22:03:37.9077498Z Running tests... 2022-03-17T22:03:37.9078962Z ---------------------------------------------------------------------- 2022-03-17T22:03:38.5370301Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108535 2022-03-17T22:03:38.5407190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108536 2022-03-17T22:03:38.5408545Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:39.6828036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:39.6829973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:39.6831273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:39.6834246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:39.6840465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:39.7843633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:41.6484797Z ok (3.740s) 2022-03-17T22:03:41.6485253Z 2022-03-17T22:03:41.6486323Z ---------------------------------------------------------------------- 2022-03-17T22:03:41.6487129Z Ran 1 test in 3.741s 2022-03-17T22:03:41.6487492Z 2022-03-17T22:03:41.6487678Z OK 2022-03-17T22:03:41.6487970Z 2022-03-17T22:03:41.6488235Z Generating XML reports... 2022-03-17T22:03:41.6555101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220337.xml 2022-03-17T22:03:44.2981060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:44.2991707Z 2022-03-17T22:03:44.2992040Z Running tests... 2022-03-17T22:03:44.2992584Z ---------------------------------------------------------------------- 2022-03-17T22:03:44.9052842Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108745 2022-03-17T22:03:44.9093391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108746 2022-03-17T22:03:44.9094773Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:46.0624925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:46.0625650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:46.0626627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:46.0627459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:46.0634651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:46.0635263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:48.0195718Z ok (3.720s) 2022-03-17T22:03:48.0196326Z 2022-03-17T22:03:48.0197367Z ---------------------------------------------------------------------- 2022-03-17T22:03:48.0198373Z Ran 1 test in 3.720s 2022-03-17T22:03:48.0198800Z 2022-03-17T22:03:48.0199066Z OK 2022-03-17T22:03:48.0199435Z 2022-03-17T22:03:48.0199791Z Generating XML reports... 2022-03-17T22:03:48.0270981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220344.xml 2022-03-17T22:03:50.6368986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:50.6377197Z 2022-03-17T22:03:50.6377811Z Running tests... 2022-03-17T22:03:50.6379149Z ---------------------------------------------------------------------- 2022-03-17T22:03:51.2420802Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108955 2022-03-17T22:03:51.2453185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108956 2022-03-17T22:03:51.2454744Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:52.4066107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:52.4067678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:52.4070110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:52.4169257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:52.4187917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:52.5181639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:52.5307433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:03:52.5309506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:52.5310827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:03:52.5312570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:54.3532567Z ok (3.715s) 2022-03-17T22:03:54.3533027Z 2022-03-17T22:03:54.3534172Z ---------------------------------------------------------------------- 2022-03-17T22:03:54.3534932Z Ran 1 test in 3.715s 2022-03-17T22:03:54.3535296Z 2022-03-17T22:03:54.3535487Z OK 2022-03-17T22:03:54.3535800Z 2022-03-17T22:03:54.3536081Z Generating XML reports... 2022-03-17T22:03:54.3614878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220350.xml 2022-03-17T22:03:57.0050314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:03:57.0063527Z 2022-03-17T22:03:57.0064097Z Running tests... 2022-03-17T22:03:57.0065485Z ---------------------------------------------------------------------- 2022-03-17T22:03:57.6288609Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109171 2022-03-17T22:03:57.6341818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109172 2022-03-17T22:03:57.6343250Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:03:58.7959066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:03:58.7960059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:58.7960645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:03:58.7961416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:03:58.7976272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:03:58.8974469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:03:58.9179683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:03:58.9180247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:03:58.9181073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:03:58.9281484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:04:01.4456111Z ok (4.439s) 2022-03-17T22:04:01.4456527Z 2022-03-17T22:04:01.4457538Z ---------------------------------------------------------------------- 2022-03-17T22:04:01.4458309Z Ran 1 test in 4.439s 2022-03-17T22:04:01.4458667Z 2022-03-17T22:04:01.4458833Z OK 2022-03-17T22:04:01.4459126Z 2022-03-17T22:04:01.4459403Z Generating XML reports... 2022-03-17T22:04:01.4542450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220357.xml 2022-03-17T22:04:04.1643022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:04.1655078Z 2022-03-17T22:04:04.1655407Z Running tests... 2022-03-17T22:04:04.1656596Z ---------------------------------------------------------------------- 2022-03-17T22:04:04.7750120Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109387 2022-03-17T22:04:04.7790140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109388 2022-03-17T22:04:04.7791512Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:05.9400836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:05.9403660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:05.9405059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:05.9406799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:05.9513963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:06.0421211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:18.5061442Z ok (14.340s) 2022-03-17T22:04:18.5061810Z 2022-03-17T22:04:18.5062739Z ---------------------------------------------------------------------- 2022-03-17T22:04:18.5063546Z Ran 1 test in 14.340s 2022-03-17T22:04:18.5063909Z 2022-03-17T22:04:18.5064076Z OK 2022-03-17T22:04:18.5064358Z 2022-03-17T22:04:18.5064631Z Generating XML reports... 2022-03-17T22:04:18.5142500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220404.xml 2022-03-17T22:04:21.4856015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:21.4866783Z 2022-03-17T22:04:21.4867336Z Running tests... 2022-03-17T22:04:21.4868539Z ---------------------------------------------------------------------- 2022-03-17T22:04:22.1147934Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109601 2022-03-17T22:04:22.1192201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109602 2022-03-17T22:04:22.1193702Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:23.2723084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:23.2724307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:23.2726137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:23.2727299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:23.2833033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:23.3747727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:25.1278332Z ok (3.641s) 2022-03-17T22:04:25.1278833Z 2022-03-17T22:04:25.1279852Z ---------------------------------------------------------------------- 2022-03-17T22:04:25.1280705Z Ran 1 test in 3.641s 2022-03-17T22:04:25.1281114Z 2022-03-17T22:04:25.1281329Z OK 2022-03-17T22:04:25.1281659Z 2022-03-17T22:04:25.1282164Z Generating XML reports... 2022-03-17T22:04:25.1366506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220421.xml 2022-03-17T22:04:27.7480901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:27.7495016Z 2022-03-17T22:04:27.7495487Z Running tests... 2022-03-17T22:04:27.7496653Z ---------------------------------------------------------------------- 2022-03-17T22:04:28.3529687Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109811 2022-03-17T22:04:28.3571067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109812 2022-03-17T22:04:28.3572434Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:29.4846138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:29.4861161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:29.4862096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:29.4950274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:29.4959934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:29.5880821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:31.3648248Z ok (3.615s) 2022-03-17T22:04:31.3648638Z 2022-03-17T22:04:31.3649614Z ---------------------------------------------------------------------- 2022-03-17T22:04:31.3650432Z Ran 1 test in 3.616s 2022-03-17T22:04:31.3650826Z 2022-03-17T22:04:31.3650997Z OK 2022-03-17T22:04:31.3651305Z 2022-03-17T22:04:31.3651584Z Generating XML reports... 2022-03-17T22:04:31.3733992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220427.xml 2022-03-17T22:04:33.9990358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:34.0000724Z 2022-03-17T22:04:34.0001099Z Running tests... 2022-03-17T22:04:34.0001627Z ---------------------------------------------------------------------- 2022-03-17T22:04:34.6041715Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110021 2022-03-17T22:04:34.6084819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110022 2022-03-17T22:04:34.6086144Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:35.7397225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:35.7398471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:35.7400593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:35.7402891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:35.7404219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:35.7405395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:37.6185680Z ok (3.618s) 2022-03-17T22:04:37.6186080Z 2022-03-17T22:04:37.6187051Z ---------------------------------------------------------------------- 2022-03-17T22:04:37.6187800Z Ran 1 test in 3.618s 2022-03-17T22:04:37.6188162Z 2022-03-17T22:04:37.6188346Z OK 2022-03-17T22:04:37.6188641Z 2022-03-17T22:04:37.6188938Z Generating XML reports... 2022-03-17T22:04:37.6261799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220433.xml 2022-03-17T22:04:40.2724204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:40.2734458Z 2022-03-17T22:04:40.2734799Z Running tests... 2022-03-17T22:04:40.2735339Z ---------------------------------------------------------------------- 2022-03-17T22:04:40.8789506Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110237 2022-03-17T22:04:40.8825219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110238 2022-03-17T22:04:40.8826791Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:42.0086457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:42.0087927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:42.0089901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:42.0091658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:42.0195250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:42.1097048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:43.9912441Z ok (3.717s) 2022-03-17T22:04:43.9912882Z 2022-03-17T22:04:43.9913947Z ---------------------------------------------------------------------- 2022-03-17T22:04:43.9914921Z Ran 1 test in 3.718s 2022-03-17T22:04:43.9915366Z 2022-03-17T22:04:43.9915591Z OK 2022-03-17T22:04:43.9915960Z 2022-03-17T22:04:43.9916249Z Generating XML reports... 2022-03-17T22:04:43.9985254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220440.xml 2022-03-17T22:04:46.7154938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:46.7164120Z 2022-03-17T22:04:46.7164512Z Running tests... 2022-03-17T22:04:46.7165884Z ---------------------------------------------------------------------- 2022-03-17T22:04:47.3196278Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110453 2022-03-17T22:04:47.3239070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110454 2022-03-17T22:04:47.3240646Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:48.4348094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:48.4349575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:48.4351593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:48.4451511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:48.4461641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:48.5465847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:50.1331891Z skip (3.416s) 2022-03-17T22:04:50.1332353Z 2022-03-17T22:04:50.1333403Z ---------------------------------------------------------------------- 2022-03-17T22:04:50.1334339Z Ran 1 test in 3.417s 2022-03-17T22:04:50.1334800Z 2022-03-17T22:04:50.1335058Z OK (skipped=1) 2022-03-17T22:04:50.1335427Z 2022-03-17T22:04:50.1335703Z Generating XML reports... 2022-03-17T22:04:50.1420961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220446.xml 2022-03-17T22:04:52.7361810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:52.7374435Z 2022-03-17T22:04:52.7374623Z Running tests... 2022-03-17T22:04:52.7375294Z ---------------------------------------------------------------------- 2022-03-17T22:04:53.3407503Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110663 2022-03-17T22:04:53.3445044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110664 2022-03-17T22:04:53.3446342Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:04:54.5054519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:04:54.5055761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:04:54.5057567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:54.5059323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:04:54.5163237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:04:54.6067810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:04:56.2528379Z skip (3.515s) 2022-03-17T22:04:56.2528843Z 2022-03-17T22:04:56.2529880Z ---------------------------------------------------------------------- 2022-03-17T22:04:56.2530818Z Ran 1 test in 3.515s 2022-03-17T22:04:56.2531239Z 2022-03-17T22:04:56.2531521Z OK (skipped=1) 2022-03-17T22:04:56.2532095Z 2022-03-17T22:04:56.2532415Z Generating XML reports... 2022-03-17T22:04:56.2609874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220452.xml 2022-03-17T22:04:58.8953751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:04:58.8963566Z 2022-03-17T22:04:58.8963849Z Running tests... 2022-03-17T22:04:58.8964969Z ---------------------------------------------------------------------- 2022-03-17T22:04:59.5031376Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110873 2022-03-17T22:04:59.5062698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110874 2022-03-17T22:04:59.5064018Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:05:00.6588523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:05:00.6589998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:00.6590962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:05:00.6591798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:00.6696011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:05:00.7605887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:05:02.4146583Z skip (3.518s) 2022-03-17T22:05:02.4147096Z 2022-03-17T22:05:02.4148225Z ---------------------------------------------------------------------- 2022-03-17T22:05:02.4149204Z Ran 1 test in 3.518s 2022-03-17T22:05:02.4149713Z 2022-03-17T22:05:02.4150112Z OK (skipped=1) 2022-03-17T22:05:02.4150949Z 2022-03-17T22:05:02.4151242Z Generating XML reports... 2022-03-17T22:05:02.4223328Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220458.xml 2022-03-17T22:05:05.0289183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:05.0302293Z 2022-03-17T22:05:05.0302724Z Running tests... 2022-03-17T22:05:05.0304049Z ---------------------------------------------------------------------- 2022-03-17T22:05:05.6499062Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111083 2022-03-17T22:05:05.6552597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111084 2022-03-17T22:05:05.6553958Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:05:06.8452550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:05:06.8453868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:05:06.8455825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:06.8457651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:06.8568617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:05:06.9468699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:05:08.9794929Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-03-17T22:05:10.9680368Z ok (5.937s) 2022-03-17T22:05:10.9683795Z 2022-03-17T22:05:10.9685046Z ---------------------------------------------------------------------- 2022-03-17T22:05:10.9685897Z Ran 1 test in 5.938s 2022-03-17T22:05:10.9686291Z 2022-03-17T22:05:10.9686477Z OK 2022-03-17T22:05:10.9686780Z 2022-03-17T22:05:10.9687058Z Generating XML reports... 2022-03-17T22:05:10.9766371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220505.xml 2022-03-17T22:05:13.6652228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:13.6665366Z 2022-03-17T22:05:13.6665726Z Running tests... 2022-03-17T22:05:13.6666357Z ---------------------------------------------------------------------- 2022-03-17T22:05:14.3111820Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111293 2022-03-17T22:05:14.3141388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111294 2022-03-17T22:05:14.3142986Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:05:15.4590810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:05:15.4592234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:05:15.4594417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:15.4692982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:15.4694601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:05:15.5616349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:05:15.5826023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:05:15.5826584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:05:15.5828455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:05:15.5837901Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-03-17T22:05:15.5929448Z No ranks successfully processed in monitoredBarrier. 2022-03-17T22:05:15.5931171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:05:15.6023222Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-03-17T22:05:17.7247770Z ok (4.058s) 2022-03-17T22:05:17.7249863Z 2022-03-17T22:05:17.7250717Z ---------------------------------------------------------------------- 2022-03-17T22:05:17.7251525Z Ran 1 test in 4.058s 2022-03-17T22:05:17.7251902Z 2022-03-17T22:05:17.7252107Z OK 2022-03-17T22:05:17.7252403Z 2022-03-17T22:05:17.7252663Z Generating XML reports... 2022-03-17T22:05:17.7335426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220513.xml 2022-03-17T22:05:20.3074792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:20.3087822Z 2022-03-17T22:05:20.3088602Z Running tests... 2022-03-17T22:05:20.3089401Z ---------------------------------------------------------------------- 2022-03-17T22:05:20.9101565Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111509 2022-03-17T22:05:20.9147081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111510 2022-03-17T22:05:20.9148441Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:05:22.0536770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:05:22.0537988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:05:22.0539777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:22.0541528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:22.0651487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:05:22.1560180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:05:22.1670781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:05:22.1672296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:05:22.1674096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:05:22.1675880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:05:22.2920236Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T22:05:23.9231682Z ok (3.614s) 2022-03-17T22:05:23.9233666Z 2022-03-17T22:05:23.9234605Z ---------------------------------------------------------------------- 2022-03-17T22:05:23.9235464Z Ran 1 test in 3.614s 2022-03-17T22:05:23.9235869Z 2022-03-17T22:05:23.9236051Z OK 2022-03-17T22:05:23.9236344Z 2022-03-17T22:05:23.9236615Z Generating XML reports... 2022-03-17T22:05:23.9303274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220520.xml 2022-03-17T22:05:26.4840995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:26.4848890Z 2022-03-17T22:05:26.4849198Z Running tests... 2022-03-17T22:05:26.4850501Z ---------------------------------------------------------------------- 2022-03-17T22:05:27.0847897Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111725 2022-03-17T22:05:27.0886011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111726 2022-03-17T22:05:27.0887582Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:05:28.2346630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:05:28.2348926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:28.2350477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:05:28.2352360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:28.2456001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:05:28.3372629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:05:29.9968138Z skip (3.511s) 2022-03-17T22:05:29.9969252Z 2022-03-17T22:05:29.9970323Z ---------------------------------------------------------------------- 2022-03-17T22:05:29.9971204Z Ran 1 test in 3.512s 2022-03-17T22:05:29.9971710Z 2022-03-17T22:05:29.9972049Z OK (skipped=1) 2022-03-17T22:05:29.9972411Z 2022-03-17T22:05:29.9972665Z Generating XML reports... 2022-03-17T22:05:30.0050529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220526.xml 2022-03-17T22:05:32.6017879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:32.6032284Z 2022-03-17T22:05:32.6032735Z Running tests... 2022-03-17T22:05:32.6033705Z ---------------------------------------------------------------------- 2022-03-17T22:05:32.6060672Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T22:05:32.6061085Z 2022-03-17T22:05:32.6061769Z ---------------------------------------------------------------------- 2022-03-17T22:05:32.6062290Z Ran 1 test in 0.003s 2022-03-17T22:05:32.6062500Z 2022-03-17T22:05:32.6062625Z OK (skipped=1) 2022-03-17T22:05:32.6062834Z 2022-03-17T22:05:32.6062983Z Generating XML reports... 2022-03-17T22:05:32.6103316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220532.xml 2022-03-17T22:05:35.2716648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:35.2726123Z 2022-03-17T22:05:35.2726472Z Running tests... 2022-03-17T22:05:35.2726992Z ---------------------------------------------------------------------- 2022-03-17T22:05:35.2751431Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T22:05:35.2751858Z 2022-03-17T22:05:35.2752476Z ---------------------------------------------------------------------- 2022-03-17T22:05:35.2753066Z Ran 1 test in 0.003s 2022-03-17T22:05:35.2753257Z 2022-03-17T22:05:35.2753404Z OK (skipped=1) 2022-03-17T22:05:35.2753571Z 2022-03-17T22:05:35.2753705Z Generating XML reports... 2022-03-17T22:05:35.2789748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220535.xml 2022-03-17T22:05:37.7913410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:37.7927994Z 2022-03-17T22:05:37.7928265Z Running tests... 2022-03-17T22:05:37.7928953Z ---------------------------------------------------------------------- 2022-03-17T22:05:37.7950022Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:05:37.7950517Z 2022-03-17T22:05:37.7951152Z ---------------------------------------------------------------------- 2022-03-17T22:05:37.7951597Z Ran 1 test in 0.002s 2022-03-17T22:05:37.7951805Z 2022-03-17T22:05:37.7951938Z OK (skipped=1) 2022-03-17T22:05:37.7952134Z 2022-03-17T22:05:37.7952297Z Generating XML reports... 2022-03-17T22:05:37.7989026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220537.xml 2022-03-17T22:05:40.3040327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:40.3058373Z 2022-03-17T22:05:40.3058726Z Running tests... 2022-03-17T22:05:40.3059331Z ---------------------------------------------------------------------- 2022-03-17T22:05:40.3081460Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T22:05:40.3081995Z 2022-03-17T22:05:40.3082499Z ---------------------------------------------------------------------- 2022-03-17T22:05:40.3082933Z Ran 1 test in 0.003s 2022-03-17T22:05:40.3083135Z 2022-03-17T22:05:40.3083272Z OK (skipped=1) 2022-03-17T22:05:40.3083455Z 2022-03-17T22:05:40.3083604Z Generating XML reports... 2022-03-17T22:05:40.3121107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220540.xml 2022-03-17T22:05:42.7868109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:42.7880307Z 2022-03-17T22:05:42.7880929Z Running tests... 2022-03-17T22:05:42.7882472Z ---------------------------------------------------------------------- 2022-03-17T22:05:42.7903405Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:05:42.7904110Z 2022-03-17T22:05:42.7905097Z ---------------------------------------------------------------------- 2022-03-17T22:05:42.7906027Z Ran 1 test in 0.002s 2022-03-17T22:05:42.7906486Z 2022-03-17T22:05:42.7906723Z OK (skipped=1) 2022-03-17T22:05:42.7907082Z 2022-03-17T22:05:42.7907358Z Generating XML reports... 2022-03-17T22:05:42.7942485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220542.xml 2022-03-17T22:05:45.4707857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:45.4718547Z 2022-03-17T22:05:45.4718975Z Running tests... 2022-03-17T22:05:45.4720068Z ---------------------------------------------------------------------- 2022-03-17T22:05:45.4738539Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:05:45.4739312Z 2022-03-17T22:05:45.4740008Z ---------------------------------------------------------------------- 2022-03-17T22:05:45.4740483Z Ran 1 test in 0.002s 2022-03-17T22:05:45.4740700Z 2022-03-17T22:05:45.4740830Z OK (skipped=1) 2022-03-17T22:05:45.4741040Z 2022-03-17T22:05:45.4741187Z Generating XML reports... 2022-03-17T22:05:45.4777535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220545.xml 2022-03-17T22:05:48.0693690Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:48.0705749Z 2022-03-17T22:05:48.0706046Z Running tests... 2022-03-17T22:05:48.0706604Z ---------------------------------------------------------------------- 2022-03-17T22:05:48.0728097Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:05:48.0728517Z 2022-03-17T22:05:48.0729179Z ---------------------------------------------------------------------- 2022-03-17T22:05:48.0729563Z Ran 1 test in 0.002s 2022-03-17T22:05:48.0729806Z 2022-03-17T22:05:48.0730720Z OK (skipped=1) 2022-03-17T22:05:48.0730893Z 2022-03-17T22:05:48.0731319Z Generating XML reports... 2022-03-17T22:05:48.0770235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220548.xml 2022-03-17T22:05:50.5749838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:50.5762066Z 2022-03-17T22:05:50.5762374Z Running tests... 2022-03-17T22:05:50.5762963Z ---------------------------------------------------------------------- 2022-03-17T22:05:50.5782040Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:05:50.5782502Z 2022-03-17T22:05:50.5782950Z ---------------------------------------------------------------------- 2022-03-17T22:05:50.5783367Z Ran 1 test in 0.002s 2022-03-17T22:05:50.5783575Z 2022-03-17T22:05:50.5783682Z OK (skipped=1) 2022-03-17T22:05:50.5783836Z 2022-03-17T22:05:50.5783960Z Generating XML reports... 2022-03-17T22:05:50.5821637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220550.xml 2022-03-17T22:05:53.0926247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:53.0939438Z 2022-03-17T22:05:53.0939838Z Running tests... 2022-03-17T22:05:53.0940408Z ---------------------------------------------------------------------- 2022-03-17T22:05:53.7337364Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112471 2022-03-17T22:05:53.7375150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112472 2022-03-17T22:05:53.7376497Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:05:54.8885682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:05:54.8886998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:05:54.8889606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:54.8987631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:05:54.9006002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:05:54.9998205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:05:56.6455162Z ok (3.551s) 2022-03-17T22:05:56.6455512Z 2022-03-17T22:05:56.6456966Z ---------------------------------------------------------------------- 2022-03-17T22:05:56.6457946Z Ran 1 test in 3.552s 2022-03-17T22:05:56.6458319Z 2022-03-17T22:05:56.6458493Z OK 2022-03-17T22:05:56.6458786Z 2022-03-17T22:05:56.6459071Z Generating XML reports... 2022-03-17T22:05:56.6539386Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220553.xml 2022-03-17T22:05:59.2083021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:05:59.2091434Z 2022-03-17T22:05:59.2091689Z Running tests... 2022-03-17T22:05:59.2092911Z ---------------------------------------------------------------------- 2022-03-17T22:05:59.8078490Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112681 2022-03-17T22:05:59.8113503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112682 2022-03-17T22:05:59.8114838Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:06:00.9573304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:06:00.9574367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:00.9574998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:06:00.9575741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:00.9685025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:06:01.0594003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:06:02.7204328Z ok (3.511s) 2022-03-17T22:06:02.7204883Z 2022-03-17T22:06:02.7205888Z ---------------------------------------------------------------------- 2022-03-17T22:06:02.7206837Z Ran 1 test in 3.511s 2022-03-17T22:06:02.7207281Z 2022-03-17T22:06:02.7207535Z OK 2022-03-17T22:06:02.7207879Z 2022-03-17T22:06:02.7208199Z Generating XML reports... 2022-03-17T22:06:02.7286284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220559.xml 2022-03-17T22:06:05.2868651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:06:05.2883042Z 2022-03-17T22:06:05.2883566Z Running tests... 2022-03-17T22:06:05.2884587Z ---------------------------------------------------------------------- 2022-03-17T22:06:05.2902936Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:06:05.2903850Z 2022-03-17T22:06:05.2904528Z ---------------------------------------------------------------------- 2022-03-17T22:06:05.2905113Z Ran 1 test in 0.002s 2022-03-17T22:06:05.2905387Z 2022-03-17T22:06:05.2905561Z OK (skipped=1) 2022-03-17T22:06:05.2905812Z 2022-03-17T22:06:05.2906010Z Generating XML reports... 2022-03-17T22:06:05.2945141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220605.xml 2022-03-17T22:06:08.0481121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:06:08.0495513Z 2022-03-17T22:06:08.0495839Z Running tests... 2022-03-17T22:06:08.0496389Z ---------------------------------------------------------------------- 2022-03-17T22:06:08.0514896Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:06:08.0515272Z 2022-03-17T22:06:08.0515759Z ---------------------------------------------------------------------- 2022-03-17T22:06:08.0516202Z Ran 1 test in 0.002s 2022-03-17T22:06:08.0516408Z 2022-03-17T22:06:08.0516543Z OK (skipped=1) 2022-03-17T22:06:08.0516736Z 2022-03-17T22:06:08.0516979Z Generating XML reports... 2022-03-17T22:06:08.0558383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220608.xml 2022-03-17T22:06:10.7131065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:06:10.7140081Z 2022-03-17T22:06:10.7140343Z Running tests... 2022-03-17T22:06:10.7140868Z ---------------------------------------------------------------------- 2022-03-17T22:06:11.3211153Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113025 2022-03-17T22:06:11.3259231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113026 2022-03-17T22:06:11.3260541Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:06:12.4687103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:06:12.4687648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:06:12.4688769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:12.4689583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:12.4697497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:06:12.4698042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:06:24.9535610Z ok (14.239s) 2022-03-17T22:06:24.9536116Z 2022-03-17T22:06:24.9538208Z ---------------------------------------------------------------------- 2022-03-17T22:06:24.9539316Z Ran 1 test in 14.239s 2022-03-17T22:06:24.9539778Z 2022-03-17T22:06:24.9540088Z OK 2022-03-17T22:06:24.9540475Z 2022-03-17T22:06:24.9540737Z Generating XML reports... 2022-03-17T22:06:24.9619940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220610.xml 2022-03-17T22:06:27.5327983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:06:27.5340081Z 2022-03-17T22:06:27.5340454Z Running tests... 2022-03-17T22:06:27.5340994Z ---------------------------------------------------------------------- 2022-03-17T22:06:28.1638529Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113239 2022-03-17T22:06:28.1675528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113240 2022-03-17T22:06:28.1677240Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:06:29.3234982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:06:29.3235764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:06:29.3236831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:29.3336287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:29.3347334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:06:29.4347931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:06:42.0023063Z ok (14.468s) 2022-03-17T22:06:42.0023829Z 2022-03-17T22:06:42.0025398Z ---------------------------------------------------------------------- 2022-03-17T22:06:42.0026449Z Ran 1 test in 14.468s 2022-03-17T22:06:42.0026911Z 2022-03-17T22:06:42.0028022Z OK 2022-03-17T22:06:42.0028411Z 2022-03-17T22:06:42.0028789Z Generating XML reports... 2022-03-17T22:06:42.0106331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220627.xml 2022-03-17T22:06:44.6054699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:06:44.6064360Z 2022-03-17T22:06:44.6064891Z Running tests... 2022-03-17T22:06:44.6066071Z ---------------------------------------------------------------------- 2022-03-17T22:06:45.2253934Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113453 2022-03-17T22:06:45.2290712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113454 2022-03-17T22:06:45.2292037Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:06:46.3612340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:06:46.3613963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:06:46.3615829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:46.3617590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:46.3620644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:06:46.3621820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:06:48.9387455Z ok (4.332s) 2022-03-17T22:06:48.9387854Z 2022-03-17T22:06:48.9389773Z ---------------------------------------------------------------------- 2022-03-17T22:06:48.9390593Z Ran 1 test in 4.332s 2022-03-17T22:06:48.9390979Z 2022-03-17T22:06:48.9391168Z OK 2022-03-17T22:06:48.9391466Z 2022-03-17T22:06:48.9391747Z Generating XML reports... 2022-03-17T22:06:48.9422167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220644.xml 2022-03-17T22:06:51.6252145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:06:51.6264012Z 2022-03-17T22:06:51.6264378Z Running tests... 2022-03-17T22:06:51.6265038Z ---------------------------------------------------------------------- 2022-03-17T22:06:52.2273132Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113663 2022-03-17T22:06:52.2308847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113664 2022-03-17T22:06:52.2310422Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:06:53.3847505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:06:53.3848781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:06:53.3850616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:53.3852392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:06:53.3860407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:06:53.3861553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:03.0568217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:07:03.0569745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:07:03.0615515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:07:03.0617043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T22:07:05.8617477Z ok (14.235s) 2022-03-17T22:07:05.8617966Z 2022-03-17T22:07:05.8619133Z ---------------------------------------------------------------------- 2022-03-17T22:07:05.8620096Z Ran 1 test in 14.235s 2022-03-17T22:07:05.8620583Z 2022-03-17T22:07:05.8623592Z OK 2022-03-17T22:07:05.8623999Z 2022-03-17T22:07:05.8624372Z Generating XML reports... 2022-03-17T22:07:05.8700316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220651.xml 2022-03-17T22:07:08.4285620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:08.4299664Z 2022-03-17T22:07:08.4299958Z Running tests... 2022-03-17T22:07:08.4300390Z ---------------------------------------------------------------------- 2022-03-17T22:07:09.0297712Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113877 2022-03-17T22:07:09.0350411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113878 2022-03-17T22:07:09.0351551Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:10.1389528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:10.1390597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:10.1392211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:10.1491663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:10.1514116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:10.2502953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:10.2708762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:07:10.2709791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:07:10.2711305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:10.2812603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:12.1447721Z ok (3.714s) 2022-03-17T22:07:12.1448184Z 2022-03-17T22:07:12.1449393Z ---------------------------------------------------------------------- 2022-03-17T22:07:12.1450739Z Ran 1 test in 3.715s 2022-03-17T22:07:12.1451184Z 2022-03-17T22:07:12.1451424Z OK 2022-03-17T22:07:12.1451886Z 2022-03-17T22:07:12.1453466Z Generating XML reports... 2022-03-17T22:07:12.1529094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220708.xml 2022-03-17T22:07:14.7798068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:14.7809464Z 2022-03-17T22:07:14.7809964Z Running tests... 2022-03-17T22:07:14.7810935Z ---------------------------------------------------------------------- 2022-03-17T22:07:15.3778587Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114099 2022-03-17T22:07:15.3810666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114100 2022-03-17T22:07:15.3812043Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:16.5012215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:16.5012789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:16.5013741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:16.5014543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:16.5124596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:16.6025857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:16.6144063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:07:16.6146349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:16.6148046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:07:16.6149824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:18.5909485Z ok (3.809s) 2022-03-17T22:07:18.5910036Z 2022-03-17T22:07:18.5911180Z ---------------------------------------------------------------------- 2022-03-17T22:07:18.5912154Z Ran 1 test in 3.810s 2022-03-17T22:07:18.5912545Z 2022-03-17T22:07:18.5912729Z OK 2022-03-17T22:07:18.5913037Z 2022-03-17T22:07:18.5913315Z Generating XML reports... 2022-03-17T22:07:18.5982783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220714.xml 2022-03-17T22:07:21.2670490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:21.2681239Z 2022-03-17T22:07:21.2681811Z Running tests... 2022-03-17T22:07:21.2683376Z ---------------------------------------------------------------------- 2022-03-17T22:07:21.8640346Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114321 2022-03-17T22:07:21.8674963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114322 2022-03-17T22:07:21.8676125Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:22.9930104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:22.9931121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:22.9932076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:22.9932893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:23.0035617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:23.0939968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:23.1144983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:07:23.1146468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:07:23.1148477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:23.1150247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:24.9776050Z ok (3.709s) 2022-03-17T22:07:24.9776665Z 2022-03-17T22:07:24.9777792Z ---------------------------------------------------------------------- 2022-03-17T22:07:24.9778759Z Ran 1 test in 3.709s 2022-03-17T22:07:24.9779202Z 2022-03-17T22:07:24.9779433Z OK 2022-03-17T22:07:24.9779824Z 2022-03-17T22:07:24.9780111Z Generating XML reports... 2022-03-17T22:07:24.9858036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220721.xml 2022-03-17T22:07:27.5831760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:27.5843378Z 2022-03-17T22:07:27.5844209Z Running tests... 2022-03-17T22:07:27.5845691Z ---------------------------------------------------------------------- 2022-03-17T22:07:28.1837791Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114543 2022-03-17T22:07:28.1883569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114544 2022-03-17T22:07:28.1886139Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:29.2784835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:29.2786100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:29.2788082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:29.2789868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:29.2894151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:29.3797343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:29.3909951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:07:29.3911225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:07:29.3913042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:29.3914821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:07:31.0973639Z ok (3.512s) 2022-03-17T22:07:31.0974041Z 2022-03-17T22:07:31.0974964Z ---------------------------------------------------------------------- 2022-03-17T22:07:31.0976297Z Ran 1 test in 3.513s 2022-03-17T22:07:31.0976672Z 2022-03-17T22:07:31.0976881Z OK 2022-03-17T22:07:31.0977181Z 2022-03-17T22:07:31.0977441Z Generating XML reports... 2022-03-17T22:07:31.1059908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220727.xml 2022-03-17T22:07:33.8073669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:33.8086031Z 2022-03-17T22:07:33.8086425Z Running tests... 2022-03-17T22:07:33.8086962Z ---------------------------------------------------------------------- 2022-03-17T22:07:34.4493292Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114765 2022-03-17T22:07:34.4553124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114766 2022-03-17T22:07:34.4554487Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:35.5775184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:35.5776661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:35.5778741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:35.5780546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:35.5882496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:35.6787298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:37.3644464Z skip (3.555s) 2022-03-17T22:07:37.3644897Z 2022-03-17T22:07:37.3646047Z ---------------------------------------------------------------------- 2022-03-17T22:07:37.3647069Z Ran 1 test in 3.556s 2022-03-17T22:07:37.3647518Z 2022-03-17T22:07:37.3647828Z OK (skipped=1) 2022-03-17T22:07:37.3648256Z 2022-03-17T22:07:37.3648601Z Generating XML reports... 2022-03-17T22:07:37.3727727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220733.xml 2022-03-17T22:07:39.9505783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:39.9514346Z 2022-03-17T22:07:39.9514472Z Running tests... 2022-03-17T22:07:39.9515120Z ---------------------------------------------------------------------- 2022-03-17T22:07:40.5503859Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114975 2022-03-17T22:07:40.5537532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114976 2022-03-17T22:07:40.5538877Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:41.6925171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:41.6926670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:41.6928829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:41.6930659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:41.6936474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:41.6940242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:43.4630370Z skip (3.511s) 2022-03-17T22:07:43.4630949Z 2022-03-17T22:07:43.4632614Z ---------------------------------------------------------------------- 2022-03-17T22:07:43.4633611Z Ran 1 test in 3.511s 2022-03-17T22:07:43.4634192Z 2022-03-17T22:07:43.4634427Z OK (skipped=1) 2022-03-17T22:07:43.4634795Z 2022-03-17T22:07:43.4635071Z Generating XML reports... 2022-03-17T22:07:43.4708014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220739.xml 2022-03-17T22:07:46.0793636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:46.0807779Z 2022-03-17T22:07:46.0808060Z Running tests... 2022-03-17T22:07:46.6774230Z ---------------------------------------------------------------------- 2022-03-17T22:07:46.6774842Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115185 2022-03-17T22:07:46.6822714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115186 2022-03-17T22:07:46.6824116Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:47.8220946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:47.8222637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:47.8224933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:47.8226835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:47.8334684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:47.9242129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:49.6945819Z skip (3.614s) 2022-03-17T22:07:49.6946543Z 2022-03-17T22:07:49.6947815Z ---------------------------------------------------------------------- 2022-03-17T22:07:49.6949091Z Ran 1 test in 3.614s 2022-03-17T22:07:49.6981365Z 2022-03-17T22:07:49.6981706Z OK (skipped=1) 2022-03-17T22:07:49.6982202Z 2022-03-17T22:07:49.6982561Z Generating XML reports... 2022-03-17T22:07:49.7022545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220746.xml 2022-03-17T22:07:52.3988761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:52.4001134Z 2022-03-17T22:07:52.4001468Z Running tests... 2022-03-17T22:07:52.4003117Z ---------------------------------------------------------------------- 2022-03-17T22:07:53.0376115Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115395 2022-03-17T22:07:53.0427895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115396 2022-03-17T22:07:53.0429288Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:07:54.1473800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:07:54.1474677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:54.1475367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:07:54.1476202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:07:54.1491486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:07:54.2486637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:07:55.8510527Z skip (3.450s) 2022-03-17T22:07:55.8510917Z 2022-03-17T22:07:55.8512178Z ---------------------------------------------------------------------- 2022-03-17T22:07:55.8512970Z Ran 1 test in 3.451s 2022-03-17T22:07:55.8513359Z 2022-03-17T22:07:55.8513583Z OK (skipped=1) 2022-03-17T22:07:55.8513943Z 2022-03-17T22:07:55.8514217Z Generating XML reports... 2022-03-17T22:07:55.8588557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220752.xml 2022-03-17T22:07:58.5141665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:07:58.5155448Z 2022-03-17T22:07:58.5155819Z Running tests... 2022-03-17T22:07:58.5156381Z ---------------------------------------------------------------------- 2022-03-17T22:07:59.1427797Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115605 2022-03-17T22:07:59.1462869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115606 2022-03-17T22:07:59.1464285Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:00.2891215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:00.2892518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:00.2894468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:00.2896306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:00.2904101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:00.3908816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:02.2545125Z ok (3.738s) 2022-03-17T22:08:02.2545590Z 2022-03-17T22:08:02.2546820Z ---------------------------------------------------------------------- 2022-03-17T22:08:02.2547606Z Ran 1 test in 3.739s 2022-03-17T22:08:02.2547964Z 2022-03-17T22:08:02.2548144Z OK 2022-03-17T22:08:02.2548434Z 2022-03-17T22:08:02.2548688Z Generating XML reports... 2022-03-17T22:08:02.2622601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220758.xml 2022-03-17T22:08:04.8173440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:04.8185025Z 2022-03-17T22:08:04.8185586Z Running tests... 2022-03-17T22:08:04.8186237Z ---------------------------------------------------------------------- 2022-03-17T22:08:05.4231384Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115821 2022-03-17T22:08:05.4266121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115822 2022-03-17T22:08:05.4267438Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:06.6079866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:06.6080829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:06.6081427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:06.6082781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:06.6194662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:06.7104479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:08.6350651Z ok (3.816s) 2022-03-17T22:08:08.6351310Z 2022-03-17T22:08:08.6352712Z ---------------------------------------------------------------------- 2022-03-17T22:08:08.6353501Z Ran 1 test in 3.816s 2022-03-17T22:08:08.6353862Z 2022-03-17T22:08:08.6354043Z OK 2022-03-17T22:08:08.6354347Z 2022-03-17T22:08:08.6354617Z Generating XML reports... 2022-03-17T22:08:08.6423055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220804.xml 2022-03-17T22:08:11.2525838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:11.2538226Z 2022-03-17T22:08:11.2538673Z Running tests... 2022-03-17T22:08:11.2539815Z ---------------------------------------------------------------------- 2022-03-17T22:08:11.2557820Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:08:11.2558602Z 2022-03-17T22:08:11.2559738Z ---------------------------------------------------------------------- 2022-03-17T22:08:11.2560550Z Ran 1 test in 0.002s 2022-03-17T22:08:11.2560926Z 2022-03-17T22:08:11.2561153Z OK (skipped=1) 2022-03-17T22:08:11.2561529Z 2022-03-17T22:08:11.2561792Z Generating XML reports... 2022-03-17T22:08:11.2598158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220811.xml 2022-03-17T22:08:13.8632474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:13.8646244Z 2022-03-17T22:08:13.8646514Z Running tests... 2022-03-17T22:08:13.8646968Z ---------------------------------------------------------------------- 2022-03-17T22:08:14.4590998Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116104 2022-03-17T22:08:14.4625155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116105 2022-03-17T22:08:14.4626472Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:15.5769549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:15.5770973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:15.5772979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:15.5872995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:15.5874915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:15.6781455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:18.0736662Z ok (4.209s) 2022-03-17T22:08:18.0737434Z 2022-03-17T22:08:18.0738841Z ---------------------------------------------------------------------- 2022-03-17T22:08:18.0739635Z Ran 1 test in 4.209s 2022-03-17T22:08:18.0739999Z 2022-03-17T22:08:18.0740203Z OK 2022-03-17T22:08:18.0740491Z 2022-03-17T22:08:18.0740747Z Generating XML reports... 2022-03-17T22:08:18.0817373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220813.xml 2022-03-17T22:08:20.7715440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:20.7725697Z 2022-03-17T22:08:20.7726377Z Running tests... 2022-03-17T22:08:20.7727851Z ---------------------------------------------------------------------- 2022-03-17T22:08:21.3713096Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116320 2022-03-17T22:08:21.3753775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116321 2022-03-17T22:08:21.3755787Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:22.5236076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:22.5237090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:22.5237766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:22.5238877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:22.5351754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:22.6258012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:24.5841118Z ok (3.811s) 2022-03-17T22:08:24.5841780Z 2022-03-17T22:08:24.5842991Z ---------------------------------------------------------------------- 2022-03-17T22:08:24.5843799Z Ran 1 test in 3.811s 2022-03-17T22:08:24.5844175Z 2022-03-17T22:08:24.5844371Z OK 2022-03-17T22:08:24.5844679Z 2022-03-17T22:08:24.5844959Z Generating XML reports... 2022-03-17T22:08:24.5923646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220820.xml 2022-03-17T22:08:27.1477668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:27.1484439Z 2022-03-17T22:08:27.1484649Z Running tests... 2022-03-17T22:08:27.1485139Z ---------------------------------------------------------------------- 2022-03-17T22:08:27.1507025Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:08:27.1507778Z 2022-03-17T22:08:27.1508742Z ---------------------------------------------------------------------- 2022-03-17T22:08:27.1509576Z Ran 1 test in 0.002s 2022-03-17T22:08:27.1509981Z 2022-03-17T22:08:27.1510212Z OK (skipped=1) 2022-03-17T22:08:27.1510555Z 2022-03-17T22:08:27.1510811Z Generating XML reports... 2022-03-17T22:08:27.1545243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220827.xml 2022-03-17T22:08:29.7563946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:29.7572181Z 2022-03-17T22:08:29.7572734Z Running tests... 2022-03-17T22:08:29.7573257Z ---------------------------------------------------------------------- 2022-03-17T22:08:29.7594635Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:08:29.7594936Z 2022-03-17T22:08:29.7595289Z ---------------------------------------------------------------------- 2022-03-17T22:08:29.7595638Z Ran 1 test in 0.002s 2022-03-17T22:08:29.7595806Z 2022-03-17T22:08:29.7595909Z OK (skipped=1) 2022-03-17T22:08:29.7596061Z 2022-03-17T22:08:29.7596207Z Generating XML reports... 2022-03-17T22:08:29.7635171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220829.xml 2022-03-17T22:08:32.3479179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:32.3496528Z 2022-03-17T22:08:32.3497435Z Running tests... 2022-03-17T22:08:32.3498615Z ---------------------------------------------------------------------- 2022-03-17T22:08:32.9872915Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116670 2022-03-17T22:08:32.9902330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116671 2022-03-17T22:08:32.9903688Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:34.1191302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:34.1193592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:34.1194892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:34.1196548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:34.1299218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:34.2209307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:35.9984911Z ok (3.649s) 2022-03-17T22:08:35.9985443Z 2022-03-17T22:08:35.9986501Z ---------------------------------------------------------------------- 2022-03-17T22:08:35.9987488Z Ran 1 test in 3.649s 2022-03-17T22:08:35.9987963Z 2022-03-17T22:08:35.9988901Z OK 2022-03-17T22:08:35.9989265Z 2022-03-17T22:08:35.9989935Z Generating XML reports... 2022-03-17T22:08:36.0064438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220832.xml 2022-03-17T22:08:38.5913846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:38.5927424Z 2022-03-17T22:08:38.5928062Z Running tests... 2022-03-17T22:08:38.5929241Z ---------------------------------------------------------------------- 2022-03-17T22:08:39.2210851Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116886 2022-03-17T22:08:39.2247863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116887 2022-03-17T22:08:39.2249237Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:40.3924220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:40.3924809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:40.3925746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:40.3926890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:40.3934751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:40.4938591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:42.2324129Z ok (3.639s) 2022-03-17T22:08:42.2324508Z 2022-03-17T22:08:42.2325390Z ---------------------------------------------------------------------- 2022-03-17T22:08:42.2326173Z Ran 1 test in 3.640s 2022-03-17T22:08:42.2326556Z 2022-03-17T22:08:42.2326746Z OK 2022-03-17T22:08:42.2327077Z 2022-03-17T22:08:42.2327355Z Generating XML reports... 2022-03-17T22:08:42.2398784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220838.xml 2022-03-17T22:08:44.8091008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:44.8099975Z 2022-03-17T22:08:44.8100338Z Running tests... 2022-03-17T22:08:44.8100875Z ---------------------------------------------------------------------- 2022-03-17T22:08:45.4107441Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117102 2022-03-17T22:08:45.4139336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117103 2022-03-17T22:08:45.4139953Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:46.5754389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:46.5756787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:46.5758131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:46.5759894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:46.5864256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:46.6780965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:48.4233897Z ok (3.613s) 2022-03-17T22:08:48.4234816Z 2022-03-17T22:08:48.4236175Z ---------------------------------------------------------------------- 2022-03-17T22:08:48.4237014Z Ran 1 test in 3.613s 2022-03-17T22:08:48.4237380Z 2022-03-17T22:08:48.4237592Z OK 2022-03-17T22:08:48.4237897Z 2022-03-17T22:08:48.4238174Z Generating XML reports... 2022-03-17T22:08:48.4311329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220844.xml 2022-03-17T22:08:51.0650809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:51.0662493Z 2022-03-17T22:08:51.0662854Z Running tests... 2022-03-17T22:08:51.0663396Z ---------------------------------------------------------------------- 2022-03-17T22:08:51.6835814Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117312 2022-03-17T22:08:51.6871909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117313 2022-03-17T22:08:51.6873237Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:08:52.8272688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:08:52.8274005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:08:52.8275929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:52.8278224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:08:52.8280296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:08:52.8281523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:08:54.5956210Z ok (3.529s) 2022-03-17T22:08:54.5956776Z 2022-03-17T22:08:54.5957936Z ---------------------------------------------------------------------- 2022-03-17T22:08:54.5958884Z Ran 1 test in 3.529s 2022-03-17T22:08:54.5959334Z 2022-03-17T22:08:54.5959608Z OK 2022-03-17T22:08:54.5960013Z 2022-03-17T22:08:54.5960338Z Generating XML reports... 2022-03-17T22:08:54.6038452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220851.xml 2022-03-17T22:08:57.2448396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:57.2460401Z 2022-03-17T22:08:57.2460783Z Running tests... 2022-03-17T22:08:57.2462025Z ---------------------------------------------------------------------- 2022-03-17T22:08:57.2476813Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:08:57.2477506Z 2022-03-17T22:08:57.2478359Z ---------------------------------------------------------------------- 2022-03-17T22:08:57.2479274Z Ran 1 test in 0.002s 2022-03-17T22:08:57.2479843Z 2022-03-17T22:08:57.2480094Z OK (skipped=1) 2022-03-17T22:08:57.2480493Z 2022-03-17T22:08:57.2480793Z Generating XML reports... 2022-03-17T22:08:57.2517598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220857.xml 2022-03-17T22:08:59.8168504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:08:59.8180769Z 2022-03-17T22:08:59.8181110Z Running tests... 2022-03-17T22:08:59.8181657Z ---------------------------------------------------------------------- 2022-03-17T22:08:59.8196224Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:08:59.8197042Z 2022-03-17T22:08:59.8197980Z ---------------------------------------------------------------------- 2022-03-17T22:08:59.8198937Z Ran 1 test in 0.002s 2022-03-17T22:08:59.8199363Z 2022-03-17T22:08:59.8199592Z OK (skipped=1) 2022-03-17T22:08:59.8199922Z 2022-03-17T22:08:59.8200186Z Generating XML reports... 2022-03-17T22:08:59.8236839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220859.xml 2022-03-17T22:09:02.3781728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:02.3794157Z 2022-03-17T22:09:02.3794437Z Running tests... 2022-03-17T22:09:02.3794870Z ---------------------------------------------------------------------- 2022-03-17T22:09:02.9780811Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117662 2022-03-17T22:09:02.9818209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117663 2022-03-17T22:09:02.9819946Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:04.1487960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:04.1489136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:04.1491031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:04.1492886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:04.1598053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:04.2505657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:04.2716475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T22:09:04.2718415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T22:09:04.2720592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:09:04.2722666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T22:09:06.0949067Z ok (3.715s) 2022-03-17T22:09:06.0949677Z 2022-03-17T22:09:06.0950685Z ---------------------------------------------------------------------- 2022-03-17T22:09:06.0951610Z Ran 1 test in 3.715s 2022-03-17T22:09:06.0952084Z 2022-03-17T22:09:06.0952275Z OK 2022-03-17T22:09:06.0952573Z 2022-03-17T22:09:06.0952845Z Generating XML reports... 2022-03-17T22:09:06.1026362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220902.xml 2022-03-17T22:09:08.6826184Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:08.6838259Z 2022-03-17T22:09:08.6838481Z Running tests... 2022-03-17T22:09:08.6838918Z ---------------------------------------------------------------------- 2022-03-17T22:09:09.2812651Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117884 2022-03-17T22:09:09.2860967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117885 2022-03-17T22:09:09.2862382Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:10.4048581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:10.4049856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:10.4051692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:10.4053465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:10.4159939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:10.5064977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:12.2935959Z skip (3.609s) 2022-03-17T22:09:12.2936397Z 2022-03-17T22:09:12.2937446Z ---------------------------------------------------------------------- 2022-03-17T22:09:12.2938409Z Ran 1 test in 3.610s 2022-03-17T22:09:12.2938863Z 2022-03-17T22:09:12.2939149Z OK (skipped=1) 2022-03-17T22:09:12.2942767Z 2022-03-17T22:09:12.2944892Z Generating XML reports... 2022-03-17T22:09:12.3017378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220908.xml 2022-03-17T22:09:14.9210192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:14.9221778Z 2022-03-17T22:09:14.9222071Z Running tests... 2022-03-17T22:09:14.9223260Z ---------------------------------------------------------------------- 2022-03-17T22:09:15.5208167Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118094 2022-03-17T22:09:15.5244637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118095 2022-03-17T22:09:15.5246003Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:16.6919966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:16.6921789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:16.6924007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:16.6926661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:16.6927970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:16.6929111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:18.5341847Z ok (3.611s) 2022-03-17T22:09:18.5342226Z 2022-03-17T22:09:18.5343174Z ---------------------------------------------------------------------- 2022-03-17T22:09:18.5343930Z Ran 1 test in 3.612s 2022-03-17T22:09:18.5344291Z 2022-03-17T22:09:18.5344474Z OK 2022-03-17T22:09:18.5344764Z 2022-03-17T22:09:18.5345060Z Generating XML reports... 2022-03-17T22:09:18.5418835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220914.xml 2022-03-17T22:09:21.1793026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:21.1807912Z 2022-03-17T22:09:21.1808614Z Running tests... 2022-03-17T22:09:21.1809339Z ---------------------------------------------------------------------- 2022-03-17T22:09:21.8105865Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118304 2022-03-17T22:09:21.8140120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118305 2022-03-17T22:09:21.8142693Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:22.9694464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:22.9695230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:22.9696236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:22.9697041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:22.9809427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:23.0715613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:24.8238850Z ok (3.643s) 2022-03-17T22:09:24.8239410Z 2022-03-17T22:09:24.8240435Z ---------------------------------------------------------------------- 2022-03-17T22:09:24.8241385Z Ran 1 test in 3.643s 2022-03-17T22:09:24.8241817Z 2022-03-17T22:09:24.8242176Z OK 2022-03-17T22:09:24.8242476Z 2022-03-17T22:09:24.8242752Z Generating XML reports... 2022-03-17T22:09:24.8324427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220921.xml 2022-03-17T22:09:27.4109575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:27.4119823Z 2022-03-17T22:09:27.4120173Z Running tests... 2022-03-17T22:09:27.4120709Z ---------------------------------------------------------------------- 2022-03-17T22:09:28.0530743Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118514 2022-03-17T22:09:28.0568119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118515 2022-03-17T22:09:28.0569450Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:29.2118343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:29.2119160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:29.2120051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:29.2219461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:29.2228560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:29.3230827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:31.3663809Z ok (3.954s) 2022-03-17T22:09:31.3664603Z 2022-03-17T22:09:31.3665924Z ---------------------------------------------------------------------- 2022-03-17T22:09:31.3666918Z Ran 1 test in 3.954s 2022-03-17T22:09:31.3667405Z 2022-03-17T22:09:31.3667655Z OK 2022-03-17T22:09:31.3667959Z 2022-03-17T22:09:31.3668266Z Generating XML reports... 2022-03-17T22:09:31.3743719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220927.xml 2022-03-17T22:09:34.0735786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:34.0746073Z 2022-03-17T22:09:34.0746531Z Running tests... 2022-03-17T22:09:34.0747956Z ---------------------------------------------------------------------- 2022-03-17T22:09:34.6722639Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118724 2022-03-17T22:09:34.6756954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118725 2022-03-17T22:09:34.6758514Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:35.7975927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:35.7977417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:35.7979482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:35.7981346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:35.7988916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:35.7990364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:37.5854395Z ok (3.510s) 2022-03-17T22:09:37.5854897Z 2022-03-17T22:09:37.5855948Z ---------------------------------------------------------------------- 2022-03-17T22:09:37.5857029Z Ran 1 test in 3.510s 2022-03-17T22:09:37.5857451Z 2022-03-17T22:09:37.5857648Z OK 2022-03-17T22:09:37.5857944Z 2022-03-17T22:09:37.5858237Z Generating XML reports... 2022-03-17T22:09:37.5935403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220934.xml 2022-03-17T22:09:40.2114956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:40.2127440Z 2022-03-17T22:09:40.2128025Z Running tests... 2022-03-17T22:09:40.2129200Z ---------------------------------------------------------------------- 2022-03-17T22:09:40.8376491Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118940 2022-03-17T22:09:40.8415670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118941 2022-03-17T22:09:40.8417057Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:41.9980892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:41.9981527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:41.9983093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:42.0081714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:42.0100553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:42.1092889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:43.9492627Z ok (3.736s) 2022-03-17T22:09:43.9493020Z 2022-03-17T22:09:43.9494009Z ---------------------------------------------------------------------- 2022-03-17T22:09:43.9494809Z Ran 1 test in 3.736s 2022-03-17T22:09:43.9495182Z 2022-03-17T22:09:43.9495366Z OK 2022-03-17T22:09:43.9495694Z 2022-03-17T22:09:43.9495979Z Generating XML reports... 2022-03-17T22:09:43.9575600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220940.xml 2022-03-17T22:09:46.5360954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:46.5372783Z 2022-03-17T22:09:46.5373138Z Running tests... 2022-03-17T22:09:46.5373702Z ---------------------------------------------------------------------- 2022-03-17T22:09:47.1355574Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119156 2022-03-17T22:09:47.1390139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119157 2022-03-17T22:09:47.1391520Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:09:48.2881508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:09:48.2882533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:48.2883124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:09:48.2883865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:09:48.2990075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:09:48.3898174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:09:50.1486295Z ok (3.611s) 2022-03-17T22:09:50.1486681Z 2022-03-17T22:09:50.1487508Z ---------------------------------------------------------------------- 2022-03-17T22:09:50.1488262Z Ran 1 test in 3.611s 2022-03-17T22:09:50.1488632Z 2022-03-17T22:09:50.1488849Z OK 2022-03-17T22:09:50.1489143Z 2022-03-17T22:09:50.1489399Z Generating XML reports... 2022-03-17T22:09:50.1566710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220946.xml 2022-03-17T22:09:52.7479701Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:52.7490083Z 2022-03-17T22:09:52.7490362Z Running tests... 2022-03-17T22:09:52.7490936Z ---------------------------------------------------------------------- 2022-03-17T22:09:52.7505862Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.001s) 2022-03-17T22:09:52.7506216Z 2022-03-17T22:09:52.7506568Z ---------------------------------------------------------------------- 2022-03-17T22:09:52.7506923Z Ran 1 test in 0.002s 2022-03-17T22:09:52.7507090Z 2022-03-17T22:09:52.7507195Z OK (skipped=1) 2022-03-17T22:09:52.7507347Z 2022-03-17T22:09:52.7507468Z Generating XML reports... 2022-03-17T22:09:52.7545275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220952.xml 2022-03-17T22:09:55.3923684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:55.3943442Z 2022-03-17T22:09:55.3943834Z Running tests... 2022-03-17T22:09:55.3944382Z ---------------------------------------------------------------------- 2022-03-17T22:09:55.3957678Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:09:55.3958021Z 2022-03-17T22:09:55.3958379Z ---------------------------------------------------------------------- 2022-03-17T22:09:55.3958720Z Ran 1 test in 0.002s 2022-03-17T22:09:55.3958884Z 2022-03-17T22:09:55.3959003Z OK (skipped=1) 2022-03-17T22:09:55.3959157Z 2022-03-17T22:09:55.3959285Z Generating XML reports... 2022-03-17T22:09:55.4002422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220955.xml 2022-03-17T22:09:58.0741068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:09:58.0753186Z 2022-03-17T22:09:58.0753759Z Running tests... 2022-03-17T22:09:58.0754889Z ---------------------------------------------------------------------- 2022-03-17T22:09:58.0774222Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T22:09:58.0774531Z 2022-03-17T22:09:58.0775240Z ---------------------------------------------------------------------- 2022-03-17T22:09:58.0775588Z Ran 1 test in 0.002s 2022-03-17T22:09:58.0775753Z 2022-03-17T22:09:58.0775864Z OK (skipped=1) 2022-03-17T22:09:58.0776031Z 2022-03-17T22:09:58.0776152Z Generating XML reports... 2022-03-17T22:09:58.0814338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220958.xml 2022-03-17T22:10:00.5618407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:00.5629442Z 2022-03-17T22:10:00.5629913Z Running tests... 2022-03-17T22:10:00.5630471Z ---------------------------------------------------------------------- 2022-03-17T22:10:01.1754564Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119573 2022-03-17T22:10:01.1787039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119574 2022-03-17T22:10:01.1788476Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:02.3377137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:02.3378229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:02.3378928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:02.3379999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:02.3489908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:02.4401196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:04.2872848Z ok (3.724s) 2022-03-17T22:10:04.2873292Z 2022-03-17T22:10:04.2874623Z ---------------------------------------------------------------------- 2022-03-17T22:10:04.2875394Z Ran 1 test in 3.724s 2022-03-17T22:10:04.2875771Z 2022-03-17T22:10:04.2875981Z OK 2022-03-17T22:10:04.2876275Z 2022-03-17T22:10:04.2876560Z Generating XML reports... 2022-03-17T22:10:04.2942154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221000.xml 2022-03-17T22:10:06.8473001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:06.8484754Z 2022-03-17T22:10:06.8485089Z Running tests... 2022-03-17T22:10:06.8485824Z ---------------------------------------------------------------------- 2022-03-17T22:10:07.4463836Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119789 2022-03-17T22:10:07.4496362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119790 2022-03-17T22:10:07.4497740Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:08.5448185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:08.5450595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:08.5451928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:08.5453606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:08.5554007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:08.6467285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:10.4621599Z ok (3.613s) 2022-03-17T22:10:10.4622784Z 2022-03-17T22:10:10.4623935Z ---------------------------------------------------------------------- 2022-03-17T22:10:10.4624907Z Ran 1 test in 3.614s 2022-03-17T22:10:10.4625363Z 2022-03-17T22:10:10.4625557Z OK 2022-03-17T22:10:10.4625845Z 2022-03-17T22:10:10.4626117Z Generating XML reports... 2022-03-17T22:10:10.4702815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221006.xml 2022-03-17T22:10:13.0436689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:13.0447224Z 2022-03-17T22:10:13.0447826Z Running tests... 2022-03-17T22:10:13.0449238Z ---------------------------------------------------------------------- 2022-03-17T22:10:13.6447247Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119999 2022-03-17T22:10:13.6484900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120000 2022-03-17T22:10:13.6486684Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:14.8266198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:14.8267574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:14.8269711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:14.8367605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:14.8371450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:14.9291952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:16.7563992Z ok (3.711s) 2022-03-17T22:10:16.7564379Z 2022-03-17T22:10:16.7566737Z ---------------------------------------------------------------------- 2022-03-17T22:10:16.7567794Z Ran 1 test in 3.711s 2022-03-17T22:10:16.7568287Z 2022-03-17T22:10:16.7568479Z OK 2022-03-17T22:10:16.7568778Z 2022-03-17T22:10:16.7569059Z Generating XML reports... 2022-03-17T22:10:16.7643841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221013.xml 2022-03-17T22:10:19.3064356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:19.3074907Z 2022-03-17T22:10:19.3075202Z Running tests... 2022-03-17T22:10:19.3075761Z ---------------------------------------------------------------------- 2022-03-17T22:10:19.9052692Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120215 2022-03-17T22:10:19.9099605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120216 2022-03-17T22:10:19.9100912Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:21.0591175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:21.0594371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:21.0597548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:21.0696810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:21.0709013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:21.1618483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:23.8192529Z ok (4.511s) 2022-03-17T22:10:23.8192995Z 2022-03-17T22:10:23.8194319Z ---------------------------------------------------------------------- 2022-03-17T22:10:23.8195218Z Ran 1 test in 4.511s 2022-03-17T22:10:23.8195569Z 2022-03-17T22:10:23.8195753Z OK 2022-03-17T22:10:23.8196056Z 2022-03-17T22:10:23.8196331Z Generating XML reports... 2022-03-17T22:10:23.8264874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221019.xml 2022-03-17T22:10:26.4354797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:26.4366748Z 2022-03-17T22:10:26.4367042Z Running tests... 2022-03-17T22:10:26.4367566Z ---------------------------------------------------------------------- 2022-03-17T22:10:27.0326651Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120431 2022-03-17T22:10:27.0372251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120432 2022-03-17T22:10:27.0373832Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:28.1722535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:28.1723012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:28.1723772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:28.1726311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:28.1834843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:28.2747245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:30.1463950Z ok (3.709s) 2022-03-17T22:10:30.1464342Z 2022-03-17T22:10:30.1465167Z ---------------------------------------------------------------------- 2022-03-17T22:10:30.1465926Z Ran 1 test in 3.710s 2022-03-17T22:10:30.1466284Z 2022-03-17T22:10:30.1466482Z OK 2022-03-17T22:10:30.1466774Z 2022-03-17T22:10:30.1467036Z Generating XML reports... 2022-03-17T22:10:30.1540903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221026.xml 2022-03-17T22:10:32.7888786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:32.7901263Z 2022-03-17T22:10:32.7901525Z Running tests... 2022-03-17T22:10:32.7901953Z ---------------------------------------------------------------------- 2022-03-17T22:10:33.3868257Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120827 2022-03-17T22:10:33.3899963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120828 2022-03-17T22:10:33.3901465Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:34.5295110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:34.5295986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:34.5296646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:34.5297496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:34.5407271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:34.6313448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:36.7995189Z ok (4.009s) 2022-03-17T22:10:36.7995653Z 2022-03-17T22:10:36.7996636Z ---------------------------------------------------------------------- 2022-03-17T22:10:36.7997494Z Ran 1 test in 4.009s 2022-03-17T22:10:36.7997899Z 2022-03-17T22:10:36.7998089Z OK 2022-03-17T22:10:36.7998415Z 2022-03-17T22:10:36.7998719Z Generating XML reports... 2022-03-17T22:10:36.8076335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221032.xml 2022-03-17T22:10:39.3937460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:39.3948419Z 2022-03-17T22:10:39.3948995Z Running tests... 2022-03-17T22:10:39.3950184Z ---------------------------------------------------------------------- 2022-03-17T22:10:39.9945705Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121223 2022-03-17T22:10:39.9992910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121224 2022-03-17T22:10:39.9994435Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:41.1595401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:41.1596848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:41.1599014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:41.1699334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:41.1714802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:41.2712500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:53.4251846Z ok (14.030s) 2022-03-17T22:10:53.4252320Z 2022-03-17T22:10:53.4254228Z ---------------------------------------------------------------------- 2022-03-17T22:10:53.4255190Z Ran 1 test in 14.030s 2022-03-17T22:10:53.4255646Z 2022-03-17T22:10:53.4255871Z OK 2022-03-17T22:10:53.4256243Z 2022-03-17T22:10:53.4256565Z Generating XML reports... 2022-03-17T22:10:53.4316390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221039.xml 2022-03-17T22:10:56.1139426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:10:56.1159240Z 2022-03-17T22:10:56.1159671Z Running tests... 2022-03-17T22:10:56.1160185Z ---------------------------------------------------------------------- 2022-03-17T22:10:56.7415289Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121437 2022-03-17T22:10:56.7466988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121438 2022-03-17T22:10:56.7468235Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:10:57.8970221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:10:57.8971029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:10:57.8971911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:57.8972709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:10:57.9077434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:10:57.9978930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:10:59.7568414Z ok (3.641s) 2022-03-17T22:10:59.7568797Z 2022-03-17T22:10:59.7570204Z ---------------------------------------------------------------------- 2022-03-17T22:10:59.7571191Z Ran 1 test in 3.641s 2022-03-17T22:10:59.7571615Z 2022-03-17T22:10:59.7571841Z OK 2022-03-17T22:10:59.7572199Z 2022-03-17T22:10:59.7572523Z Generating XML reports... 2022-03-17T22:10:59.7651397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221056.xml 2022-03-17T22:11:02.4026628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:11:02.4040952Z 2022-03-17T22:11:02.4041267Z Running tests... 2022-03-17T22:11:02.4041787Z ---------------------------------------------------------------------- 2022-03-17T22:11:03.0026542Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121713 2022-03-17T22:11:03.0066052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121714 2022-03-17T22:11:03.0067597Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:11:04.1425737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:11:04.1426327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:11:04.1427214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:04.1428011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:04.1430072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:11:04.1430598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:11:06.1152188Z ok (3.711s) 2022-03-17T22:11:06.1152652Z 2022-03-17T22:11:06.1154273Z ---------------------------------------------------------------------- 2022-03-17T22:11:06.1155253Z Ran 1 test in 3.711s 2022-03-17T22:11:06.1155647Z 2022-03-17T22:11:06.1155876Z OK 2022-03-17T22:11:06.1156216Z 2022-03-17T22:11:06.1156533Z Generating XML reports... 2022-03-17T22:11:06.1225634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221102.xml 2022-03-17T22:11:08.7534054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:11:08.7549938Z 2022-03-17T22:11:08.7550309Z Running tests... 2022-03-17T22:11:08.7550826Z ---------------------------------------------------------------------- 2022-03-17T22:11:09.3537132Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121923 2022-03-17T22:11:09.3568722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121924 2022-03-17T22:11:09.3570417Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:11:10.4683758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:11:10.4685043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:11:10.4687009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:10.4786157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:10.4807080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:11:10.5802186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:11:19.8650364Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:11:19.9586679Z [W reducer.cpp:1256] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T22:11:22.5844609Z ok (13.829s) 2022-03-17T22:11:22.5845103Z 2022-03-17T22:11:22.5846532Z ---------------------------------------------------------------------- 2022-03-17T22:11:22.5847423Z Ran 1 test in 13.829s 2022-03-17T22:11:22.5847838Z 2022-03-17T22:11:22.5852461Z OK 2022-03-17T22:11:22.5852884Z 2022-03-17T22:11:22.5853189Z Generating XML reports... 2022-03-17T22:11:22.5924340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221108.xml 2022-03-17T22:11:25.1732842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:11:25.1746668Z 2022-03-17T22:11:25.1747087Z Running tests... 2022-03-17T22:11:25.1747636Z ---------------------------------------------------------------------- 2022-03-17T22:11:25.7997366Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122137 2022-03-17T22:11:25.8041139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122138 2022-03-17T22:11:25.8043087Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:11:26.9623656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:11:26.9624300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:11:26.9625188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:26.9625986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:26.9639446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:11:26.9640017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:11:28.7126545Z skip (3.538s) 2022-03-17T22:11:28.7127009Z 2022-03-17T22:11:28.7128264Z ---------------------------------------------------------------------- 2022-03-17T22:11:28.7129391Z Ran 1 test in 3.538s 2022-03-17T22:11:28.7129905Z 2022-03-17T22:11:28.7130234Z OK (skipped=1) 2022-03-17T22:11:28.7130650Z 2022-03-17T22:11:28.7130956Z Generating XML reports... 2022-03-17T22:11:28.7211109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221125.xml 2022-03-17T22:11:31.2671345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T22:11:31.2681261Z 2022-03-17T22:11:31.2682692Z Running tests... 2022-03-17T22:11:31.2684180Z ---------------------------------------------------------------------- 2022-03-17T22:11:31.8669248Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122347 2022-03-17T22:11:31.8700608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122348 2022-03-17T22:11:31.8702231Z [W Module.cpp:498] Warning: Disabling benchmark mode for MIOpen is NOT supported. Overriding value to True (function operator()) 2022-03-17T22:11:33.0228963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:11:33.0230047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:33.0231450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T22:11:33.0233248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T22:11:33.0243784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T22:11:33.1244232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T22:11:34.8795221Z skip (3.611s) 2022-03-17T22:11:34.8795638Z 2022-03-17T22:11:34.8797050Z ---------------------------------------------------------------------- 2022-03-17T22:11:34.8798041Z Ran 1 test in 3.611s 2022-03-17T22:11:34.8798502Z 2022-03-17T22:11:34.8798821Z OK (skipped=1) 2022-03-17T22:11:34.8799178Z 2022-03-17T22:11:34.8799457Z Generating XML reports... 2022-03-17T22:11:34.8878284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221131.xml 2022-03-17T22:11:37.8665752Z Running distributed/test_launcher ... [2022-03-17 22:11:37.865683] 2022-03-17T22:11:37.8667855Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 22:11:37.865837] 2022-03-17T22:11:38.9440769Z Test results will be stored in test-reports/dist-gloo/distributed.test_launcher 2022-03-17T22:11:38.9454247Z 2022-03-17T22:11:38.9455012Z Running tests... 2022-03-17T22:11:38.9456469Z ---------------------------------------------------------------------- 2022-03-17T22:11:39.5402708Z test_launch_user_script (__main__.TestDistributedLaunch) ... /opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated 2022-03-17T22:11:39.5404456Z and will be removed in future. Use torchrun. 2022-03-17T22:11:39.5405935Z Note that --use_env is set by default in torchrun. 2022-03-17T22:11:39.5407316Z If your script expects `--local_rank` argument to be set, please 2022-03-17T22:11:39.5408534Z change it to read from `os.environ['LOCAL_RANK']` instead. See 2022-03-17T22:11:39.5409823Z https://pytorch.org/docs/stable/distributed.html#launch-utility for 2022-03-17T22:11:39.5410700Z further instructions 2022-03-17T22:11:39.5411106Z 2022-03-17T22:11:39.5411364Z FutureWarning, 2022-03-17T22:11:39.5414690Z WARNING:torch.distributed.run: 2022-03-17T22:11:39.5415408Z ***************************************** 2022-03-17T22:11:39.5416790Z Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 2022-03-17T22:11:39.5418105Z ***************************************** 2022-03-17T22:11:39.5737949Z Success, smoke test 2022-03-17T22:11:39.5834148Z Success, smoke test 2022-03-17T22:11:39.5913442Z Success, smoke test 2022-03-17T22:11:39.6054407Z Success, smoke test 2022-03-17T22:11:40.5884532Z ok (1.643s) 2022-03-17T22:11:40.5885303Z 2022-03-17T22:11:40.5886165Z ---------------------------------------------------------------------- 2022-03-17T22:11:40.5887358Z Ran 1 test in 1.643s 2022-03-17T22:11:40.5887731Z 2022-03-17T22:11:40.5887902Z OK 2022-03-17T22:11:40.5888193Z 2022-03-17T22:11:40.5888468Z Generating XML reports... 2022-03-17T22:11:40.6122109Z Generated XML report: test-reports/dist-gloo/distributed.test_launcher/TEST-TestDistributedLaunch-20220317221138.xml 2022-03-17T22:11:42.1392345Z Running distributed/test_nccl ... [2022-03-17 22:11:42.138340] 2022-03-17T22:11:42.1394091Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 22:11:42.138513] 2022-03-17T22:11:43.6180445Z Test results will be stored in test-reports/dist-gloo/distributed.test_nccl 2022-03-17T22:11:43.6202771Z 2022-03-17T22:11:43.6203380Z Running tests... 2022-03-17T22:11:43.6204567Z ---------------------------------------------------------------------- 2022-03-17T22:11:44.1375944Z test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T22:11:44.2440339Z ok (0.623s) 2022-03-17T22:11:44.2504640Z test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.006s) 2022-03-17T22:11:44.2558084Z test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.006s) 2022-03-17T22:11:44.2614020Z test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.006s) 2022-03-17T22:11:44.2663725Z test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T22:11:44.2707891Z test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-03-17T22:11:44.2744504Z test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.003s) 2022-03-17T22:11:44.2782234Z test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-03-17T22:11:44.2820892Z test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-03-17T22:11:44.2871864Z test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T22:11:44.2945640Z test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.007s) 2022-03-17T22:11:44.2965227Z test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.002s) 2022-03-17T22:11:44.2965798Z 2022-03-17T22:11:44.2966960Z ---------------------------------------------------------------------- 2022-03-17T22:11:44.2967994Z Ran 12 tests in 0.676s 2022-03-17T22:11:44.2968370Z 2022-03-17T22:11:44.2968564Z OK 2022-03-17T22:11:44.2968846Z 2022-03-17T22:11:44.2969143Z Generating XML reports... 2022-03-17T22:11:44.3031024Z Generated XML report: test-reports/dist-gloo/distributed.test_nccl/TEST-TestNCCLCUDA-20220317221143.xml 2022-03-17T22:11:45.8756691Z Running distributed/test_pg_wrapper ... [2022-03-17 22:11:45.874713] 2022-03-17T22:11:45.8758915Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 22:11:45.874888] 2022-03-17T22:11:46.7211051Z 2022-03-17T22:11:46.7212082Z 2022-03-17T22:11:48.7694582Z Running distributed/test_store ... [2022-03-17 22:11:48.768593] 2022-03-17T22:11:48.7697613Z Executing ['/opt/conda/bin/python3.7', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 22:11:48.768769] 2022-03-17T22:11:49.6189265Z test_compare_set (__main__.FileStoreTest) 2022-03-17T22:11:49.6189825Z test_set_get (__main__.FileStoreTest) 2022-03-17T22:11:49.6190394Z test_compare_set (__main__.PrefixFileStoreTest) 2022-03-17T22:11:49.6190756Z test_set_get (__main__.PrefixFileStoreTest) 2022-03-17T22:11:49.6191155Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-03-17T22:11:49.6191635Z test_set_get (__main__.PrefixTCPStoreTest) 2022-03-17T22:11:49.6192101Z test_set_get (__main__.PythonStoreTest) 2022-03-17T22:11:49.6192655Z test_nominal (__main__.RendezvousEnvTest) 2022-03-17T22:11:49.6193032Z test_common_errors (__main__.RendezvousFileTest) 2022-03-17T22:11:49.6193420Z test_nominal (__main__.RendezvousFileTest) 2022-03-17T22:11:49.6194060Z test_unknown_handler (__main__.RendezvousTest) 2022-03-17T22:11:49.6194462Z test_address_already_in_use (__main__.TCPStoreTest) 2022-03-17T22:11:49.6194833Z test_compare_set (__main__.TCPStoreTest) 2022-03-17T22:11:49.6195244Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-03-17T22:11:49.6195686Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-03-17T22:11:49.6196131Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-03-17T22:11:49.6196541Z test_multitenancy (__main__.TCPStoreTest) 2022-03-17T22:11:49.6196916Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-03-17T22:11:49.6197279Z test_set_get (__main__.TCPStoreTest) 2022-03-17T22:11:50.4987939Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:11:50.4999955Z 2022-03-17T22:11:50.5000282Z Running tests... 2022-03-17T22:11:50.5000823Z ---------------------------------------------------------------------- 2022-03-17T22:11:51.0795601Z test_compare_set (__main__.FileStoreTest) ... ok (0.579s) 2022-03-17T22:11:51.0796230Z 2022-03-17T22:11:51.0797260Z ---------------------------------------------------------------------- 2022-03-17T22:11:51.0798219Z Ran 1 test in 0.580s 2022-03-17T22:11:51.0798582Z 2022-03-17T22:11:51.0798765Z OK 2022-03-17T22:11:51.0799057Z 2022-03-17T22:11:51.0799329Z Generating XML reports... 2022-03-17T22:11:51.0825473Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317221150.xml 2022-03-17T22:11:53.5522396Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:11:53.5535957Z 2022-03-17T22:11:53.5536671Z Running tests... 2022-03-17T22:11:53.5537988Z ---------------------------------------------------------------------- 2022-03-17T22:11:54.1372732Z test_set_get (__main__.FileStoreTest) ... ok (0.583s) 2022-03-17T22:11:54.1373294Z 2022-03-17T22:11:54.1374162Z ---------------------------------------------------------------------- 2022-03-17T22:11:54.1375015Z Ran 1 test in 0.584s 2022-03-17T22:11:54.1375415Z 2022-03-17T22:11:54.1375627Z OK 2022-03-17T22:11:54.1375953Z 2022-03-17T22:11:54.1376254Z Generating XML reports... 2022-03-17T22:11:54.1405362Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317221153.xml 2022-03-17T22:11:56.5747839Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:11:56.5761441Z 2022-03-17T22:11:56.5762109Z Running tests... 2022-03-17T22:11:56.5762616Z ---------------------------------------------------------------------- 2022-03-17T22:11:57.1550882Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.579s) 2022-03-17T22:11:57.1551443Z 2022-03-17T22:11:57.1552517Z ---------------------------------------------------------------------- 2022-03-17T22:11:57.1553492Z Ran 1 test in 0.579s 2022-03-17T22:11:57.1553976Z 2022-03-17T22:11:57.1554187Z OK 2022-03-17T22:11:57.1554493Z 2022-03-17T22:11:57.1554794Z Generating XML reports... 2022-03-17T22:11:57.1581327Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317221156.xml 2022-03-17T22:11:59.5181824Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:11:59.5194005Z 2022-03-17T22:11:59.5194259Z Running tests... 2022-03-17T22:11:59.5194683Z ---------------------------------------------------------------------- 2022-03-17T22:12:00.0967411Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.577s) 2022-03-17T22:12:00.0967621Z 2022-03-17T22:12:00.0968675Z ---------------------------------------------------------------------- 2022-03-17T22:12:00.0969191Z Ran 1 test in 0.577s 2022-03-17T22:12:00.0969420Z 2022-03-17T22:12:00.0969530Z OK 2022-03-17T22:12:00.0969734Z 2022-03-17T22:12:00.0969909Z Generating XML reports... 2022-03-17T22:12:00.0999405Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317221159.xml 2022-03-17T22:12:02.5248086Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:02.5261412Z 2022-03-17T22:12:02.5261742Z Running tests... 2022-03-17T22:12:02.5262212Z ---------------------------------------------------------------------- 2022-03-17T22:12:03.1038015Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.577s) 2022-03-17T22:12:03.1039034Z 2022-03-17T22:12:03.1040097Z ---------------------------------------------------------------------- 2022-03-17T22:12:03.1041192Z Ran 1 test in 0.578s 2022-03-17T22:12:03.1041662Z 2022-03-17T22:12:03.1042123Z OK 2022-03-17T22:12:03.1042595Z 2022-03-17T22:12:03.1042870Z Generating XML reports... 2022-03-17T22:12:03.1072648Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317221202.xml 2022-03-17T22:12:06.0814855Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:06.0826002Z 2022-03-17T22:12:06.0826498Z Running tests... 2022-03-17T22:12:06.0827752Z ---------------------------------------------------------------------- 2022-03-17T22:12:06.6619551Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.579s) 2022-03-17T22:12:06.6620527Z 2022-03-17T22:12:06.6621796Z ---------------------------------------------------------------------- 2022-03-17T22:12:06.6622794Z Ran 1 test in 0.580s 2022-03-17T22:12:06.6623285Z 2022-03-17T22:12:06.6623473Z OK 2022-03-17T22:12:06.6623773Z 2022-03-17T22:12:06.6624061Z Generating XML reports... 2022-03-17T22:12:06.6654898Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317221206.xml 2022-03-17T22:12:09.1591723Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:09.1605116Z 2022-03-17T22:12:09.1605985Z Running tests... 2022-03-17T22:12:09.1607500Z ---------------------------------------------------------------------- 2022-03-17T22:12:09.7439025Z test_set_get (__main__.PythonStoreTest) ... ok (0.583s) 2022-03-17T22:12:09.7439803Z 2022-03-17T22:12:09.7440843Z ---------------------------------------------------------------------- 2022-03-17T22:12:09.7441757Z Ran 1 test in 0.583s 2022-03-17T22:12:09.7442294Z 2022-03-17T22:12:09.7442482Z OK 2022-03-17T22:12:09.7442753Z 2022-03-17T22:12:09.7443024Z Generating XML reports... 2022-03-17T22:12:09.7472012Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PythonStoreTest-20220317221209.xml 2022-03-17T22:12:12.1459612Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:12.1471422Z 2022-03-17T22:12:12.1471664Z Running tests... 2022-03-17T22:12:12.1472190Z ---------------------------------------------------------------------- 2022-03-17T22:12:12.7258048Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.578s) 2022-03-17T22:12:12.7258598Z 2022-03-17T22:12:12.7259599Z ---------------------------------------------------------------------- 2022-03-17T22:12:12.7260568Z Ran 1 test in 0.579s 2022-03-17T22:12:12.7261034Z 2022-03-17T22:12:12.7261285Z OK 2022-03-17T22:12:12.7261598Z 2022-03-17T22:12:12.7261879Z Generating XML reports... 2022-03-17T22:12:12.7292332Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousEnvTest-20220317221212.xml 2022-03-17T22:12:15.1624139Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:15.1637725Z 2022-03-17T22:12:15.1638785Z Running tests... 2022-03-17T22:12:15.1640055Z ---------------------------------------------------------------------- 2022-03-17T22:12:15.7431664Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.579s) 2022-03-17T22:12:15.7432558Z 2022-03-17T22:12:15.7433888Z ---------------------------------------------------------------------- 2022-03-17T22:12:15.7434960Z Ran 1 test in 0.579s 2022-03-17T22:12:15.7435416Z 2022-03-17T22:12:15.7435662Z OK 2022-03-17T22:12:15.7436034Z 2022-03-17T22:12:15.7436388Z Generating XML reports... 2022-03-17T22:12:15.7464390Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317221215.xml 2022-03-17T22:12:18.1881022Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:18.1893205Z 2022-03-17T22:12:18.1893700Z Running tests... 2022-03-17T22:12:18.1894892Z ---------------------------------------------------------------------- 2022-03-17T22:12:18.7694655Z test_nominal (__main__.RendezvousFileTest) ... ok (0.580s) 2022-03-17T22:12:18.7695332Z 2022-03-17T22:12:18.7696271Z ---------------------------------------------------------------------- 2022-03-17T22:12:18.7697023Z Ran 1 test in 0.580s 2022-03-17T22:12:18.7697389Z 2022-03-17T22:12:18.7697587Z OK 2022-03-17T22:12:18.7697866Z 2022-03-17T22:12:18.7698140Z Generating XML reports... 2022-03-17T22:12:18.7726772Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317221218.xml 2022-03-17T22:12:21.2298396Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:21.2311053Z 2022-03-17T22:12:21.2311339Z Running tests... 2022-03-17T22:12:21.2311915Z ---------------------------------------------------------------------- 2022-03-17T22:12:21.8076778Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.576s) 2022-03-17T22:12:21.8077161Z 2022-03-17T22:12:21.8077863Z ---------------------------------------------------------------------- 2022-03-17T22:12:21.8078782Z Ran 1 test in 0.577s 2022-03-17T22:12:21.8079267Z 2022-03-17T22:12:21.8079486Z OK 2022-03-17T22:12:21.8079787Z 2022-03-17T22:12:21.8080073Z Generating XML reports... 2022-03-17T22:12:21.8106709Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousTest-20220317221221.xml 2022-03-17T22:12:24.3149969Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:24.3160560Z 2022-03-17T22:12:24.3161264Z Running tests... 2022-03-17T22:12:24.3162033Z ---------------------------------------------------------------------- 2022-03-17T22:12:24.8930165Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:42025 (errno: 98 - Address already in use). 2022-03-17T22:12:24.8941271Z [W socket.cpp:401] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use). 2022-03-17T22:12:24.8942274Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address. 2022-03-17T22:12:24.9085077Z ok (0.592s) 2022-03-17T22:12:24.9085253Z 2022-03-17T22:12:24.9085706Z ---------------------------------------------------------------------- 2022-03-17T22:12:24.9086152Z Ran 1 test in 0.592s 2022-03-17T22:12:24.9086364Z 2022-03-17T22:12:24.9086474Z OK 2022-03-17T22:12:24.9086660Z 2022-03-17T22:12:24.9086790Z Generating XML reports... 2022-03-17T22:12:24.9122637Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221224.xml 2022-03-17T22:12:27.5004452Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:27.5017806Z 2022-03-17T22:12:27.5018146Z Running tests... 2022-03-17T22:12:27.5019139Z ---------------------------------------------------------------------- 2022-03-17T22:12:28.0888992Z test_compare_set (__main__.TCPStoreTest) ... ok (0.587s) 2022-03-17T22:12:28.0889698Z 2022-03-17T22:12:28.0890897Z ---------------------------------------------------------------------- 2022-03-17T22:12:28.0891683Z Ran 1 test in 0.587s 2022-03-17T22:12:28.0892068Z 2022-03-17T22:12:28.0892253Z OK 2022-03-17T22:12:28.0892540Z 2022-03-17T22:12:28.0892817Z Generating XML reports... 2022-03-17T22:12:28.0921381Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221227.xml 2022-03-17T22:12:30.5545931Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:30.5558225Z 2022-03-17T22:12:30.5559064Z Running tests... 2022-03-17T22:12:30.5560245Z ---------------------------------------------------------------------- 2022-03-17T22:12:31.1512320Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T22:12:31.1514481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T22:12:31.1576398Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-03-17T22:12:31.2155882Z ok (0.660s) 2022-03-17T22:12:31.2156275Z 2022-03-17T22:12:31.2157089Z ---------------------------------------------------------------------- 2022-03-17T22:12:31.2157883Z Ran 1 test in 0.660s 2022-03-17T22:12:31.2158264Z 2022-03-17T22:12:31.2158449Z OK 2022-03-17T22:12:31.2158758Z 2022-03-17T22:12:31.2159038Z Generating XML reports... 2022-03-17T22:12:31.2227308Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221230.xml 2022-03-17T22:12:33.7031168Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:33.7043201Z 2022-03-17T22:12:33.7043954Z Running tests... 2022-03-17T22:12:33.7045166Z ---------------------------------------------------------------------- 2022-03-17T22:12:34.3319657Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.627s) 2022-03-17T22:12:34.3320304Z 2022-03-17T22:12:34.3321817Z ---------------------------------------------------------------------- 2022-03-17T22:12:34.3322950Z Ran 1 test in 0.628s 2022-03-17T22:12:34.3323335Z 2022-03-17T22:12:34.3323538Z OK 2022-03-17T22:12:34.3323856Z 2022-03-17T22:12:34.3324134Z Generating XML reports... 2022-03-17T22:12:34.3377407Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221233.xml 2022-03-17T22:12:36.7548609Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:36.7561357Z 2022-03-17T22:12:36.7561787Z Running tests... 2022-03-17T22:12:36.7563306Z ---------------------------------------------------------------------- 2022-03-17T22:12:37.3554013Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.599s) 2022-03-17T22:12:37.3554687Z 2022-03-17T22:12:37.3555511Z ---------------------------------------------------------------------- 2022-03-17T22:12:37.3556307Z Ran 1 test in 0.599s 2022-03-17T22:12:37.3556658Z 2022-03-17T22:12:37.3556851Z OK 2022-03-17T22:12:37.3557158Z 2022-03-17T22:12:37.3557845Z Generating XML reports... 2022-03-17T22:12:37.3587474Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221236.xml 2022-03-17T22:12:39.7280659Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:39.7290154Z 2022-03-17T22:12:39.7290606Z Running tests... 2022-03-17T22:12:39.7291221Z ---------------------------------------------------------------------- 2022-03-17T22:12:40.3153697Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.586s) 2022-03-17T22:12:40.3154416Z 2022-03-17T22:12:40.3155611Z ---------------------------------------------------------------------- 2022-03-17T22:12:40.3156442Z Ran 1 test in 0.586s 2022-03-17T22:12:40.3156804Z 2022-03-17T22:12:40.3156986Z OK 2022-03-17T22:12:40.3157258Z 2022-03-17T22:12:40.3157526Z Generating XML reports... 2022-03-17T22:12:40.3191929Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221239.xml 2022-03-17T22:12:42.7368864Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:42.7381483Z 2022-03-17T22:12:42.7381766Z Running tests... 2022-03-17T22:12:42.7382297Z ---------------------------------------------------------------------- 2022-03-17T22:12:45.3314343Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.593s) 2022-03-17T22:12:45.3314596Z 2022-03-17T22:12:45.3315966Z ---------------------------------------------------------------------- 2022-03-17T22:12:45.3316831Z Ran 1 test in 2.593s 2022-03-17T22:12:45.3317797Z 2022-03-17T22:12:45.3317994Z OK 2022-03-17T22:12:45.3318273Z 2022-03-17T22:12:45.3318572Z Generating XML reports... 2022-03-17T22:12:45.3348041Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221242.xml 2022-03-17T22:12:47.7594052Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T22:12:47.7604672Z 2022-03-17T22:12:47.7604923Z Running tests... 2022-03-17T22:12:47.7605467Z ---------------------------------------------------------------------- 2022-03-17T22:12:48.3379203Z test_set_get (__main__.TCPStoreTest) ... ok (0.577s) 2022-03-17T22:12:48.3379755Z 2022-03-17T22:12:48.3380747Z ---------------------------------------------------------------------- 2022-03-17T22:12:48.3381692Z Ran 1 test in 0.577s 2022-03-17T22:12:48.3382142Z 2022-03-17T22:12:48.3382377Z OK 2022-03-17T22:12:48.3382736Z 2022-03-17T22:12:48.3383107Z Generating XML reports... 2022-03-17T22:12:48.3412445Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221247.xml 2022-03-17T22:12:53.2090933Z 2022-03-17T22:12:53.2091643Z real 166m35.340s 2022-03-17T22:12:53.2092447Z user 265m32.757s 2022-03-17T22:12:53.2093109Z sys 184m9.282s 2022-03-17T22:12:53.2093845Z + assert_git_not_dirty 2022-03-17T22:12:53.2095384Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed != *rocm* ]] 2022-03-17T22:12:53.2097043Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed == *cuda* ]] 2022-03-17T22:12:53.2098092Z + test_rpc 2022-03-17T22:12:53.2099206Z + [[ linux-bionic-rocm4.5-py3.7-distributed-distributed != *rocm* ]] 2022-03-17T22:12:53.2100048Z + cleanup 2022-03-17T22:12:53.2100568Z + retcode=0 2022-03-17T22:12:53.2101105Z + set +x 2022-03-17T22:12:53.2101674Z EXITED_USER_LAND 2022-03-17T22:12:53.2175685Z + docker exec -t 38dccec0eac565a8c82df885e30ce5584a016dd65e3050f93bf07338c6b019b2 sh -c 'cd ../pytorch && sudo cp -R test/test-reports ../workspace/test' 2022-03-17T22:12:53.3883834Z ##[group]Run python3 -m pip install junitparser==2.1.1 rich==10.9.0 2022-03-17T22:12:53.3884983Z python3 -m pip install junitparser==2.1.1 rich==10.9.0 2022-03-17T22:12:53.3947509Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T22:12:53.3947993Z env: 2022-03-17T22:12:53.3948506Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:53.3949344Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:53.3950143Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:53.3950842Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:53.3951445Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:53.3951849Z IN_CI: 1 2022-03-17T22:12:53.3952199Z IS_GHA: 1 2022-03-17T22:12:53.3952580Z IN_WHEEL_TEST: 1 2022-03-17T22:12:53.3953089Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:53.3953782Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:53.3954317Z PR_LABELS: [] 2022-03-17T22:12:53.3955098Z GITHUB_TOKEN: *** 2022-03-17T22:12:53.3955546Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:53.3955961Z PR_NUMBER: 2022-03-17T22:12:53.3956407Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:53.3956875Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:53.3957336Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:53.3958118Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:53.3959023Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:53.3959582Z TEST_CONFIG: distributed 2022-03-17T22:12:53.3959985Z SHARD_NUMBER: 1 2022-03-17T22:12:53.3960380Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:53.3960763Z PR_BODY: 2022-03-17T22:12:53.3961225Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:53.3962029Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:53.3962853Z SHM_SIZE: 8g 2022-03-17T22:12:53.3963227Z ##[endgroup] 2022-03-17T22:12:54.2984281Z Collecting junitparser==2.1.1 2022-03-17T22:12:54.4868081Z Using cached https://files.pythonhosted.org/packages/2d/56/206da86c05dca67932d052deb27222aad4dd7e6c54254866aafe96fa252e/junitparser-2.1.1-py2.py3-none-any.whl 2022-03-17T22:12:54.4909181Z Collecting rich==10.9.0 2022-03-17T22:12:54.6622678Z Using cached https://files.pythonhosted.org/packages/e9/c4/ea9a6e34dd5c4c3049b0a86c49feb7f38c9f5e901ffb6eeb832e89dfa540/rich-10.9.0-py3-none-any.whl 2022-03-17T22:12:54.6849188Z Collecting future (from junitparser==2.1.1) 2022-03-17T22:12:54.7831424Z Collecting dataclasses<0.9,>=0.7; python_version >= "3.6" and python_version < "3.7" (from rich==10.9.0) 2022-03-17T22:12:54.8422579Z Using cached https://files.pythonhosted.org/packages/fe/ca/75fac5856ab5cfa51bbbcefa250182e50441074fdc3f803f6e76451fab43/dataclasses-0.8-py3-none-any.whl 2022-03-17T22:12:54.8444289Z Collecting commonmark<0.10.0,>=0.9.0 (from rich==10.9.0) 2022-03-17T22:12:54.9098396Z Using cached https://files.pythonhosted.org/packages/b1/92/dfd892312d822f36c55366118b95d914e5f16de11044a27cf10a7d71bbbf/commonmark-0.9.1-py2.py3-none-any.whl 2022-03-17T22:12:54.9177240Z Collecting pygments<3.0.0,>=2.6.0 (from rich==10.9.0) 2022-03-17T22:12:55.0287016Z Using cached https://files.pythonhosted.org/packages/1d/17/ed4d2df187995561b28f1073df24137cb750e12f9879d291cc8ab67c65d2/Pygments-2.11.2-py3-none-any.whl 2022-03-17T22:12:55.0780261Z Collecting typing-extensions<4.0.0,>=3.7.4; python_version < "3.8" (from rich==10.9.0) 2022-03-17T22:12:55.1455679Z Using cached https://files.pythonhosted.org/packages/74/60/18783336cc7fcdd95dae91d73477830aa53f5d3181ae4fe20491d7fc3199/typing_extensions-3.10.0.2-py3-none-any.whl 2022-03-17T22:12:55.1491674Z Collecting colorama<0.5.0,>=0.4.0 (from rich==10.9.0) 2022-03-17T22:12:55.2125394Z Using cached https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl 2022-03-17T22:12:55.2160075Z Installing collected packages: future, junitparser, dataclasses, commonmark, pygments, typing-extensions, colorama, rich 2022-03-17T22:12:56.1498479Z Successfully installed colorama-0.4.4 commonmark-0.9.1 dataclasses-0.8 future-0.18.2 junitparser-2.1.1 pygments-2.11.2 rich-10.9.0 typing-extensions-4.1.1 2022-03-17T22:12:56.2220357Z ##[group]Run python3 tools/render_junit.py test/ 2022-03-17T22:12:56.2221256Z python3 tools/render_junit.py test/ 2022-03-17T22:12:56.2302754Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T22:12:56.2303473Z env: 2022-03-17T22:12:56.2304275Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:56.2305523Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:56.2306741Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:56.2307851Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:56.2308780Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:56.2309399Z IN_CI: 1 2022-03-17T22:12:56.2309935Z IS_GHA: 1 2022-03-17T22:12:56.2310504Z IN_WHEEL_TEST: 1 2022-03-17T22:12:56.2311271Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:56.2312311Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:56.2313125Z PR_LABELS: [] 2022-03-17T22:12:56.2314068Z GITHUB_TOKEN: *** 2022-03-17T22:12:56.2314733Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:56.2315375Z PR_NUMBER: 2022-03-17T22:12:56.2316023Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:56.2316743Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:56.2317448Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:56.2318636Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:56.2320276Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:56.2321140Z TEST_CONFIG: distributed 2022-03-17T22:12:56.2321754Z SHARD_NUMBER: 1 2022-03-17T22:12:56.2322703Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:56.2323290Z PR_BODY: 2022-03-17T22:12:56.2323989Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:56.2325066Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:56.2326023Z SHM_SIZE: 8g 2022-03-17T22:12:56.2326670Z PYTHONIOENCODING: utf-8 2022-03-17T22:12:56.2327309Z ##[endgroup] 2022-03-17T22:12:56.4588149Z ✅ 923 Passed 2022-03-17T22:12:56.4589599Z 💨 590 Skipped 2022-03-17T22:12:56.4594031Z 🚨 0 Failed 2022-03-17T22:12:56.4840333Z ##[group]Run # Remove any previous test jsons if they exist 2022-03-17T22:12:56.4840692Z # Remove any previous test jsons if they exist 2022-03-17T22:12:56.4840970Z rm -f test-jsons-*.zip 2022-03-17T22:12:56.4841295Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2022-03-17T22:12:56.4875415Z shell: /bin/bash -e {0} 2022-03-17T22:12:56.4875624Z env: 2022-03-17T22:12:56.4875906Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:56.4876341Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:56.4876767Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:56.4877148Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:56.4877467Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:56.4877686Z IN_CI: 1 2022-03-17T22:12:56.4877875Z IS_GHA: 1 2022-03-17T22:12:56.4878076Z IN_WHEEL_TEST: 1 2022-03-17T22:12:56.4878370Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:56.4878717Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:56.4879006Z PR_LABELS: [] 2022-03-17T22:12:56.4879424Z GITHUB_TOKEN: *** 2022-03-17T22:12:56.4879660Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:56.4879880Z PR_NUMBER: 2022-03-17T22:12:56.4880121Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:56.4880380Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:56.4880621Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:56.4881046Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:56.4881522Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:56.4881892Z TEST_CONFIG: distributed 2022-03-17T22:12:56.4882148Z SHARD_NUMBER: 1 2022-03-17T22:12:56.4882364Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:56.4882579Z PR_BODY: 2022-03-17T22:12:56.4882819Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:56.4883200Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:56.4883583Z SHM_SIZE: 8g 2022-03-17T22:12:56.4883913Z FILE_SUFFIX: test_distributed_1_1-distributed-1-1-linux.rocm.gpu 2022-03-17T22:12:56.4884247Z ##[endgroup] 2022-03-17T22:12:56.5141006Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2022-03-17T22:12:56.5211897Z ##[group]Run actions/upload-artifact@v2 2022-03-17T22:12:56.5212594Z with: 2022-03-17T22:12:56.5213184Z retention-days: 14 2022-03-17T22:12:56.5213830Z if-no-files-found: warn 2022-03-17T22:12:56.5214529Z path: test-jsons-*.zip 2022-03-17T22:12:56.5215144Z name: artifact 2022-03-17T22:12:56.5215699Z env: 2022-03-17T22:12:56.5216485Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:56.5217740Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:56.5218950Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:56.5220037Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:56.5221218Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:56.5221845Z IN_CI: 1 2022-03-17T22:12:56.5222384Z IS_GHA: 1 2022-03-17T22:12:56.5222951Z IN_WHEEL_TEST: 1 2022-03-17T22:12:56.5223774Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:56.5224757Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:56.5225580Z PR_LABELS: [] 2022-03-17T22:12:56.5226535Z GITHUB_TOKEN: *** 2022-03-17T22:12:56.5227203Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:56.5227846Z PR_NUMBER: 2022-03-17T22:12:56.5228522Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:56.5229255Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:56.5229940Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:56.5231153Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:56.5232539Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:56.5233397Z TEST_CONFIG: distributed 2022-03-17T22:12:56.5234039Z SHARD_NUMBER: 1 2022-03-17T22:12:56.5234627Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:56.5235236Z PR_BODY: 2022-03-17T22:12:56.5235929Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:56.5237001Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:56.5237932Z SHM_SIZE: 8g 2022-03-17T22:12:56.5238480Z ##[endgroup] 2022-03-17T22:12:56.5971887Z With the provided path, there will be 1 file uploaded 2022-03-17T22:12:56.5973856Z Starting artifact upload 2022-03-17T22:12:56.5975979Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-03-17T22:12:56.5977451Z Artifact name is valid! 2022-03-17T22:12:57.0121001Z Container for artifact "artifact" successfully created. Starting upload of file(s) 2022-03-17T22:12:57.4283945Z Total size of all the files uploaded is 17999 bytes 2022-03-17T22:12:57.4284954Z File upload process has finished. Finalizing the artifact upload 2022-03-17T22:12:57.7819912Z Artifact has been finalized. All files have been successfully uploaded! 2022-03-17T22:12:57.7820554Z 2022-03-17T22:12:57.7821040Z The raw size of all the files that were specified for upload is 18027 bytes 2022-03-17T22:12:57.7822342Z The size of all the files that were uploaded is 17999 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-03-17T22:12:57.7823164Z 2022-03-17T22:12:57.7827443Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-03-17T22:12:57.7828512Z 2022-03-17T22:12:57.7828925Z Artifact artifact has been successfully uploaded! 2022-03-17T22:12:57.7931340Z ##[group]Run # Remove any previous test reports if they exist 2022-03-17T22:12:57.7932336Z # Remove any previous test reports if they exist 2022-03-17T22:12:57.7933126Z rm -f test-reports-*.zip 2022-03-17T22:12:57.7933954Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' 2022-03-17T22:12:57.8011970Z shell: /bin/bash -e {0} 2022-03-17T22:12:57.8012222Z env: 2022-03-17T22:12:57.8012547Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:57.8013070Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:57.8013580Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:57.8014025Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:57.8014416Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:57.8014671Z IN_CI: 1 2022-03-17T22:12:57.8014904Z IS_GHA: 1 2022-03-17T22:12:57.8015170Z IN_WHEEL_TEST: 1 2022-03-17T22:12:57.8015494Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:57.8016022Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:57.8016366Z PR_LABELS: [] 2022-03-17T22:12:57.8016772Z GITHUB_TOKEN: *** 2022-03-17T22:12:57.8017054Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:57.8017318Z PR_NUMBER: 2022-03-17T22:12:57.8017604Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:57.8017910Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:57.8018194Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:57.8018701Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:57.8019265Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:57.8019622Z TEST_CONFIG: distributed 2022-03-17T22:12:57.8019894Z SHARD_NUMBER: 1 2022-03-17T22:12:57.8020142Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:57.8020399Z PR_BODY: 2022-03-17T22:12:57.8020686Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:57.8021152Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:57.8021478Z SHM_SIZE: 8g 2022-03-17T22:12:57.8021766Z FILE_SUFFIX: test_distributed_1_1-distributed-1-1-linux.rocm.gpu 2022-03-17T22:12:57.8022047Z ##[endgroup] 2022-03-17T22:12:57.8320226Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203825.xml (deflated 40%) 2022-03-17T22:12:57.8322599Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202140.xml (deflated 43%) 2022-03-17T22:12:57.8324626Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203439.xml (deflated 42%) 2022-03-17T22:12:57.8326648Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201744.xml (deflated 41%) 2022-03-17T22:12:57.8328676Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201250.xml (deflated 41%) 2022-03-17T22:12:57.8330671Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210148.xml (deflated 40%) 2022-03-17T22:12:57.8332645Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204225.xml (deflated 41%) 2022-03-17T22:12:57.8334664Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205208.xml (deflated 42%) 2022-03-17T22:12:57.8336761Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204758.xml (deflated 42%) 2022-03-17T22:12:57.8339401Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204000.xml (deflated 46%) 2022-03-17T22:12:57.8341439Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204532.xml (deflated 42%) 2022-03-17T22:12:57.8343435Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205011.xml (deflated 43%) 2022-03-17T22:12:57.8345393Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204751.xml (deflated 41%) 2022-03-17T22:12:57.8347371Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201140.xml (deflated 44%) 2022-03-17T22:12:57.8349333Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201712.xml (deflated 41%) 2022-03-17T22:12:57.8351298Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201738.xml (deflated 40%) 2022-03-17T22:12:57.8353288Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203801.xml (deflated 41%) 2022-03-17T22:12:57.8355680Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203259.xml (deflated 43%) 2022-03-17T22:12:57.8357628Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204727.xml (deflated 41%) 2022-03-17T22:12:57.8359622Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201404.xml (deflated 41%) 2022-03-17T22:12:57.8361595Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201719.xml (deflated 42%) 2022-03-17T22:12:57.8363828Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204426.xml (deflated 43%) 2022-03-17T22:12:57.8365806Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203759.xml (deflated 42%) 2022-03-17T22:12:57.8367787Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205923.xml (deflated 42%) 2022-03-17T22:12:57.8369750Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210308.xml (deflated 42%) 2022-03-17T22:12:57.8371725Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201238.xml (deflated 42%) 2022-03-17T22:12:57.8373678Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203124.xml (deflated 43%) 2022-03-17T22:12:57.8375658Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210533.xml (deflated 41%) 2022-03-17T22:12:57.8377630Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201339.xml (deflated 42%) 2022-03-17T22:12:57.8379596Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203839.xml (deflated 40%) 2022-03-17T22:12:57.8381561Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204314.xml (deflated 42%) 2022-03-17T22:12:57.8383520Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203619.xml (deflated 43%) 2022-03-17T22:12:57.8385470Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201751.xml (deflated 42%) 2022-03-17T22:12:57.8387454Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204454.xml (deflated 42%) 2022-03-17T22:12:57.8389622Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203616.xml (deflated 43%) 2022-03-17T22:12:57.8391590Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210246.xml (deflated 40%) 2022-03-17T22:12:57.8393570Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201544.xml (deflated 41%) 2022-03-17T22:12:57.8395533Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202313.xml (deflated 41%) 2022-03-17T22:12:57.8397482Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201821.xml (deflated 42%) 2022-03-17T22:12:57.8399482Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204303.xml (deflated 42%) 2022-03-17T22:12:57.8401450Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204207.xml (deflated 40%) 2022-03-17T22:12:57.8403912Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203535.xml (deflated 43%) 2022-03-17T22:12:57.8406108Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210119.xml (deflated 41%) 2022-03-17T22:12:57.8408090Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204417.xml (deflated 42%) 2022-03-17T22:12:57.8409051Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204321.xml (deflated 42%) 2022-03-17T22:12:57.8409782Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204804.xml (deflated 42%) 2022-03-17T22:12:57.8410541Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202529.xml (deflated 41%) 2022-03-17T22:12:57.8411276Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210314.xml (deflated 41%) 2022-03-17T22:12:57.8412016Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204316.xml (deflated 43%) 2022-03-17T22:12:57.8412754Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201309.xml (deflated 41%) 2022-03-17T22:12:57.8413506Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201132.xml (deflated 44%) 2022-03-17T22:12:57.8414235Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200939.xml (deflated 44%) 2022-03-17T22:12:57.8414971Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204420.xml (deflated 43%) 2022-03-17T22:12:57.8415701Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205522.xml (deflated 42%) 2022-03-17T22:12:57.8416444Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203921.xml (deflated 43%) 2022-03-17T22:12:57.8417202Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210311.xml (deflated 41%) 2022-03-17T22:12:57.8417929Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203608.xml (deflated 43%) 2022-03-17T22:12:57.8418666Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201127.xml (deflated 42%) 2022-03-17T22:12:57.8419416Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205135.xml (deflated 42%) 2022-03-17T22:12:57.8420219Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204748.xml (deflated 44%) 2022-03-17T22:12:57.8420991Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203406.xml (deflated 42%) 2022-03-17T22:12:57.8421724Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201240.xml (deflated 42%) 2022-03-17T22:12:57.8422451Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202356.xml (deflated 43%) 2022-03-17T22:12:57.8423191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203101.xml (deflated 41%) 2022-03-17T22:12:57.8423939Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201901.xml (deflated 40%) 2022-03-17T22:12:57.8424682Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201334.xml (deflated 43%) 2022-03-17T22:12:57.8425491Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204250.xml (deflated 42%) 2022-03-17T22:12:57.8426240Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203202.xml (deflated 41%) 2022-03-17T22:12:57.8426964Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205532.xml (deflated 41%) 2022-03-17T22:12:57.8427710Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204514.xml (deflated 43%) 2022-03-17T22:12:57.8428438Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203540.xml (deflated 43%) 2022-03-17T22:12:57.8429182Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204516.xml (deflated 42%) 2022-03-17T22:12:57.8429924Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203951.xml (deflated 41%) 2022-03-17T22:12:57.8430649Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201653.xml (deflated 42%) 2022-03-17T22:12:57.8431403Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204247.xml (deflated 42%) 2022-03-17T22:12:57.8432138Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202836.xml (deflated 41%) 2022-03-17T22:12:57.8432865Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203401.xml (deflated 42%) 2022-03-17T22:12:57.8433607Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205953.xml (deflated 41%) 2022-03-17T22:12:57.8434360Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203720.xml (deflated 41%) 2022-03-17T22:12:57.8435087Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203727.xml (deflated 42%) 2022-03-17T22:12:57.8435826Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202455.xml (deflated 41%) 2022-03-17T22:12:57.8436565Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210039.xml (deflated 42%) 2022-03-17T22:12:57.8437292Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204200.xml (deflated 40%) 2022-03-17T22:12:57.8438095Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202227.xml (deflated 42%) 2022-03-17T22:12:57.8438826Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204558.xml (deflated 42%) 2022-03-17T22:12:57.8439570Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201628.xml (deflated 41%) 2022-03-17T22:12:57.8440314Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205358.xml (deflated 41%) 2022-03-17T22:12:57.8441042Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200856.xml (deflated 40%) 2022-03-17T22:12:57.8441784Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201332.xml (deflated 42%) 2022-03-17T22:12:57.8442578Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203605.xml (deflated 43%) 2022-03-17T22:12:57.8443310Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204311.xml (deflated 42%) 2022-03-17T22:12:57.8444114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205823.xml (deflated 41%) 2022-03-17T22:12:57.8444851Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202402.xml (deflated 42%) 2022-03-17T22:12:57.8445590Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203506.xml (deflated 41%) 2022-03-17T22:12:57.8446326Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204635.xml (deflated 42%) 2022-03-17T22:12:57.8447067Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204230.xml (deflated 42%) 2022-03-17T22:12:57.8447794Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204052.xml (deflated 46%) 2022-03-17T22:12:57.8448539Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205205.xml (deflated 42%) 2022-03-17T22:12:57.8449291Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202721.xml (deflated 41%) 2022-03-17T22:12:57.8450016Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210112.xml (deflated 41%) 2022-03-17T22:12:57.8450755Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210054.xml (deflated 42%) 2022-03-17T22:12:57.8451483Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205129.xml (deflated 42%) 2022-03-17T22:12:57.8452230Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202853.xml (deflated 41%) 2022-03-17T22:12:57.8452982Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203415.xml (deflated 41%) 2022-03-17T22:12:57.8453705Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203156.xml (deflated 42%) 2022-03-17T22:12:57.8454455Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201313.xml (deflated 42%) 2022-03-17T22:12:57.8455194Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204411.xml (deflated 43%) 2022-03-17T22:12:57.8455920Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201444.xml (deflated 43%) 2022-03-17T22:12:57.8456726Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204901.xml (deflated 41%) 2022-03-17T22:12:57.8457485Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204931.xml (deflated 42%) 2022-03-17T22:12:57.8458219Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203048.xml (deflated 41%) 2022-03-17T22:12:57.8458966Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203458.xml (deflated 42%) 2022-03-17T22:12:57.8459711Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200745.xml (deflated 42%) 2022-03-17T22:12:57.8460437Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201420.xml (deflated 43%) 2022-03-17T22:12:57.8461197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201202.xml (deflated 42%) 2022-03-17T22:12:57.8462007Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201603.xml (deflated 42%) 2022-03-17T22:12:57.8462731Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203423.xml (deflated 42%) 2022-03-17T22:12:57.8463469Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210428.xml (deflated 44%) 2022-03-17T22:12:57.8464196Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210205.xml (deflated 41%) 2022-03-17T22:12:57.8464950Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201537.xml (deflated 41%) 2022-03-17T22:12:57.8465700Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203639.xml (deflated 41%) 2022-03-17T22:12:57.8466430Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203151.xml (deflated 42%) 2022-03-17T22:12:57.8467180Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200743.xml (deflated 42%) 2022-03-17T22:12:57.8467923Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201349.xml (deflated 42%) 2022-03-17T22:12:57.8468644Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205335.xml (deflated 42%) 2022-03-17T22:12:57.8469382Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201135.xml (deflated 44%) 2022-03-17T22:12:57.8470116Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202028.xml (deflated 43%) 2022-03-17T22:12:57.8470845Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200811.xml (deflated 43%) 2022-03-17T22:12:57.8471599Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204721.xml (deflated 40%) 2022-03-17T22:12:57.8472335Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204141.xml (deflated 42%) 2022-03-17T22:12:57.8473063Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205432.xml (deflated 41%) 2022-03-17T22:12:57.8473801Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203005.xml (deflated 42%) 2022-03-17T22:12:57.8474529Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203159.xml (deflated 42%) 2022-03-17T22:12:57.8475333Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204604.xml (deflated 43%) 2022-03-17T22:12:57.8476085Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204504.xml (deflated 42%) 2022-03-17T22:12:57.8476812Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210340.xml (deflated 42%) 2022-03-17T22:12:57.8477551Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202048.xml (deflated 41%) 2022-03-17T22:12:57.8478299Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210436.xml (deflated 43%) 2022-03-17T22:12:57.8479024Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203042.xml (deflated 40%) 2022-03-17T22:12:57.8479766Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202630.xml (deflated 41%) 2022-03-17T22:12:57.8480560Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201516.xml (deflated 41%) 2022-03-17T22:12:57.8481285Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201009.xml (deflated 42%) 2022-03-17T22:12:57.8482092Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204844.xml (deflated 40%) 2022-03-17T22:12:57.8482843Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210407.xml (deflated 41%) 2022-03-17T22:12:57.8483575Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204544.xml (deflated 42%) 2022-03-17T22:12:57.8484316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201159.xml (deflated 42%) 2022-03-17T22:12:57.8485065Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205047.xml (deflated 41%) 2022-03-17T22:12:57.8485799Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202511.xml (deflated 41%) 2022-03-17T22:12:57.8486533Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205946.xml (deflated 40%) 2022-03-17T22:12:57.8487257Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202011.xml (deflated 41%) 2022-03-17T22:12:57.8487993Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201245.xml (deflated 41%) 2022-03-17T22:12:57.8488736Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203107.xml (deflated 41%) 2022-03-17T22:12:57.8489468Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201329.xml (deflated 42%) 2022-03-17T22:12:57.8490216Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201218.xml (deflated 40%) 2022-03-17T22:12:57.8490952Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204232.xml (deflated 41%) 2022-03-17T22:12:57.8491677Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203154.xml (deflated 42%) 2022-03-17T22:12:57.8492412Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210509.xml (deflated 43%) 2022-03-17T22:12:57.8493222Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204607.xml (deflated 42%) 2022-03-17T22:12:57.8493966Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203336.xml (deflated 41%) 2022-03-17T22:12:57.8494709Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204542.xml (deflated 42%) 2022-03-17T22:12:57.8495445Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210543.xml (deflated 41%) 2022-03-17T22:12:57.8496171Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203622.xml (deflated 41%) 2022-03-17T22:12:57.8496921Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201827.xml (deflated 41%) 2022-03-17T22:12:57.8497644Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201255.xml (deflated 42%) 2022-03-17T22:12:57.8498386Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204457.xml (deflated 41%) 2022-03-17T22:12:57.8499198Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202208.xml (deflated 42%) 2022-03-17T22:12:57.8499934Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203957.xml (deflated 42%) 2022-03-17T22:12:57.8500679Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203641.xml (deflated 40%) 2022-03-17T22:12:57.8501413Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204242.xml (deflated 42%) 2022-03-17T22:12:57.8502137Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201753.xml (deflated 41%) 2022-03-17T22:12:57.8502885Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203520.xml (deflated 42%) 2022-03-17T22:12:57.8503641Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204035.xml (deflated 48%) 2022-03-17T22:12:57.8504373Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210125.xml (deflated 41%) 2022-03-17T22:12:57.8505114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204507.xml (deflated 42%) 2022-03-17T22:12:57.8505854Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201322.xml (deflated 42%) 2022-03-17T22:12:57.8506581Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202349.xml (deflated 42%) 2022-03-17T22:12:57.8507333Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200814.xml (deflated 42%) 2022-03-17T22:12:57.8508079Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201422.xml (deflated 43%) 2022-03-17T22:12:57.8508802Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204745.xml (deflated 42%) 2022-03-17T22:12:57.8509547Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202123.xml (deflated 41%) 2022-03-17T22:12:57.8510274Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203142.xml (deflated 43%) 2022-03-17T22:12:57.8511014Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201143.xml (deflated 42%) 2022-03-17T22:12:57.8511829Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210356.xml (deflated 42%) 2022-03-17T22:12:57.8512571Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204346.xml (deflated 41%) 2022-03-17T22:12:57.8513310Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201844.xml (deflated 41%) 2022-03-17T22:12:57.8514057Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204535.xml (deflated 41%) 2022-03-17T22:12:57.8514784Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210349.xml (deflated 41%) 2022-03-17T22:12:57.8515523Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210154.xml (deflated 41%) 2022-03-17T22:12:57.8516271Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205449.xml (deflated 41%) 2022-03-17T22:12:57.8517057Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201211.xml (deflated 40%) 2022-03-17T22:12:57.8517806Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201324.xml (deflated 42%) 2022-03-17T22:12:57.8518541Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204401.xml (deflated 42%) 2022-03-17T22:12:57.8519274Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202233.xml (deflated 42%) 2022-03-17T22:12:57.8520024Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204125.xml (deflated 42%) 2022-03-17T22:12:57.8520772Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201941.xml (deflated 40%) 2022-03-17T22:12:57.8521498Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203215.xml (deflated 41%) 2022-03-17T22:12:57.8522365Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201525.xml (deflated 42%) 2022-03-17T22:12:57.8523095Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200816.xml (deflated 41%) 2022-03-17T22:12:57.8523845Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205252.xml (deflated 40%) 2022-03-17T22:12:57.8524602Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210100.xml (deflated 42%) 2022-03-17T22:12:57.8525326Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203244.xml (deflated 40%) 2022-03-17T22:12:57.8526068Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204217.xml (deflated 41%) 2022-03-17T22:12:57.8526815Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202238.xml (deflated 41%) 2022-03-17T22:12:57.8527539Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203055.xml (deflated 40%) 2022-03-17T22:12:57.8528291Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205157.xml (deflated 42%) 2022-03-17T22:12:57.8529042Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205213.xml (deflated 42%) 2022-03-17T22:12:57.8529771Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205816.xml (deflated 41%) 2022-03-17T22:12:57.8530609Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210322.xml (deflated 42%) 2022-03-17T22:12:57.8531359Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205215.xml (deflated 42%) 2022-03-17T22:12:57.8532083Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204253.xml (deflated 42%) 2022-03-17T22:12:57.8532831Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202657.xml (deflated 41%) 2022-03-17T22:12:57.8533574Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205235.xml (deflated 41%) 2022-03-17T22:12:57.8534301Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201316.xml (deflated 42%) 2022-03-17T22:12:57.8535044Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204235.xml (deflated 41%) 2022-03-17T22:12:57.8535841Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210606.xml (deflated 42%) 2022-03-17T22:12:57.8536592Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210145.xml (deflated 42%) 2022-03-17T22:12:57.8537336Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210105.xml (deflated 40%) 2022-03-17T22:12:57.8538063Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202548.xml (deflated 41%) 2022-03-17T22:12:57.8538805Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201815.xml (deflated 42%) 2022-03-17T22:12:57.8539566Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202956.xml (deflated 40%) 2022-03-17T22:12:57.8540295Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205152.xml (deflated 42%) 2022-03-17T22:12:57.8541034Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210031.xml (deflated 44%) 2022-03-17T22:12:57.8541774Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203236.xml (deflated 42%) 2022-03-17T22:12:57.8542500Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203523.xml (deflated 42%) 2022-03-17T22:12:57.8543245Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205525.xml (deflated 41%) 2022-03-17T22:12:57.8543983Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201532.xml (deflated 43%) 2022-03-17T22:12:57.8544711Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203857.xml (deflated 40%) 2022-03-17T22:12:57.8545462Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205150.xml (deflated 42%) 2022-03-17T22:12:57.8546190Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203736.xml (deflated 42%) 2022-03-17T22:12:57.8547197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210222.xml (deflated 41%) 2022-03-17T22:12:57.8547936Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210057.xml (deflated 42%) 2022-03-17T22:12:57.8548751Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201411.xml (deflated 40%) 2022-03-17T22:12:57.8549512Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204017.xml (deflated 47%) 2022-03-17T22:12:57.8550262Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201035.xml (deflated 47%) 2022-03-17T22:12:57.8550988Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203730.xml (deflated 42%) 2022-03-17T22:12:57.8551726Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210132.xml (deflated 41%) 2022-03-17T22:12:57.8552473Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210200.xml (deflated 42%) 2022-03-17T22:12:57.8553211Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203756.xml (deflated 42%) 2022-03-17T22:12:57.8553962Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204144.xml (deflated 42%) 2022-03-17T22:12:57.8554774Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205549.xml (deflated 41%) 2022-03-17T22:12:57.8555497Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203532.xml (deflated 43%) 2022-03-17T22:12:57.8556232Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204339.xml (deflated 41%) 2022-03-17T22:12:57.8556982Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203417.xml (deflated 42%) 2022-03-17T22:12:57.8557712Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203526.xml (deflated 42%) 2022-03-17T22:12:57.8558466Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205926.xml (deflated 40%) 2022-03-17T22:12:57.8559212Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202159.xml (deflated 42%) 2022-03-17T22:12:57.8559947Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204324.xml (deflated 41%) 2022-03-17T22:12:57.8560684Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204122.xml (deflated 44%) 2022-03-17T22:12:57.8561407Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201337.xml (deflated 43%) 2022-03-17T22:12:57.8562223Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210431.xml (deflated 43%) 2022-03-17T22:12:57.8562970Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202332.xml (deflated 41%) 2022-03-17T22:12:57.8563699Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210504.xml (deflated 43%) 2022-03-17T22:12:57.8564447Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202923.xml (deflated 42%) 2022-03-17T22:12:57.8565186Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201248.xml (deflated 42%) 2022-03-17T22:12:57.8565914Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203344.xml (deflated 40%) 2022-03-17T22:12:57.8566690Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202235.xml (deflated 42%) 2022-03-17T22:12:57.8567490Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201018.xml (deflated 46%) 2022-03-17T22:12:57.8568228Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204640.xml (deflated 43%) 2022-03-17T22:12:57.8568966Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204220.xml (deflated 42%) 2022-03-17T22:12:57.8569703Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201700.xml (deflated 42%) 2022-03-17T22:12:57.8570437Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205210.xml (deflated 42%) 2022-03-17T22:12:57.8571182Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202216.xml (deflated 42%) 2022-03-17T22:12:57.8571914Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202940.xml (deflated 41%) 2022-03-17T22:12:57.8572728Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205933.xml (deflated 40%) 2022-03-17T22:12:57.8573474Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201935.xml (deflated 41%) 2022-03-17T22:12:57.8574209Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201319.xml (deflated 42%) 2022-03-17T22:12:57.8574949Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205329.xml (deflated 42%) 2022-03-17T22:12:57.8575691Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205940.xml (deflated 41%) 2022-03-17T22:12:57.8576424Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210139.xml (deflated 42%) 2022-03-17T22:12:57.8577181Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205626.xml (deflated 40%) 2022-03-17T22:12:57.8577920Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205605.xml (deflated 41%) 2022-03-17T22:12:57.8578656Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203456.xml (deflated 42%) 2022-03-17T22:12:57.8579399Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201457.xml (deflated 42%) 2022-03-17T22:12:57.8580146Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203613.xml (deflated 43%) 2022-03-17T22:12:57.8580880Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205200.xml (deflated 42%) 2022-03-17T22:12:57.8581618Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201622.xml (deflated 43%) 2022-03-17T22:12:57.8582355Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203513.xml (deflated 41%) 2022-03-17T22:12:57.8583079Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204215.xml (deflated 41%) 2022-03-17T22:12:57.8583829Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204352.xml (deflated 40%) 2022-03-17T22:12:57.8584563Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202211.xml (deflated 43%) 2022-03-17T22:12:57.8585300Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202647.xml (deflated 41%) 2022-03-17T22:12:57.8586100Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204827.xml (deflated 41%) 2022-03-17T22:12:57.8586835Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203611.xml (deflated 43%) 2022-03-17T22:12:57.8587577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204300.xml (deflated 42%) 2022-03-17T22:12:57.8588315Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203229.xml (deflated 40%) 2022-03-17T22:12:57.8589045Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204127.xml (deflated 40%) 2022-03-17T22:12:57.8589787Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203420.xml (deflated 42%) 2022-03-17T22:12:57.8590534Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210455.xml (deflated 41%) 2022-03-17T22:12:57.8591331Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202420.xml (deflated 42%) 2022-03-17T22:12:57.8592068Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210319.xml (deflated 42%) 2022-03-17T22:12:57.8592810Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201235.xml (deflated 41%) 2022-03-17T22:12:57.8593536Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204336.xml (deflated 41%) 2022-03-17T22:12:57.8594277Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205846.xml (deflated 41%) 2022-03-17T22:12:57.8595003Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202650.xml (deflated 41%) 2022-03-17T22:12:57.8595744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202613.xml (deflated 41%) 2022-03-17T22:12:57.8596500Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201805.xml (deflated 44%) 2022-03-17T22:12:57.8597225Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204258.xml (deflated 42%) 2022-03-17T22:12:57.8597970Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200932.xml (deflated 40%) 2022-03-17T22:12:57.8598715Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203426.xml (deflated 42%) 2022-03-17T22:12:57.8599444Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201658.xml (deflated 43%) 2022-03-17T22:12:57.8600191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201306.xml (deflated 42%) 2022-03-17T22:12:57.8600936Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210420.xml (deflated 42%) 2022-03-17T22:12:57.8601667Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204525.xml (deflated 41%) 2022-03-17T22:12:57.8602539Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205800.xml (deflated 41%) 2022-03-17T22:12:57.8603277Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210005.xml (deflated 41%) 2022-03-17T22:12:57.8604072Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201301.xml (deflated 42%) 2022-03-17T22:12:57.8604827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202545.xml (deflated 42%) 2022-03-17T22:12:57.8605576Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204655.xml (deflated 40%) 2022-03-17T22:12:57.8606305Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205309.xml (deflated 41%) 2022-03-17T22:12:57.8607049Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204222.xml (deflated 41%) 2022-03-17T22:12:57.8607775Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204435.xml (deflated 43%) 2022-03-17T22:12:57.8608522Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201625.xml (deflated 43%) 2022-03-17T22:12:57.8609259Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203015.xml (deflated 42%) 2022-03-17T22:12:57.8610050Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210012.xml (deflated 41%) 2022-03-17T22:12:57.8610788Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210337.xml (deflated 42%) 2022-03-17T22:12:57.8611523Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202031.xml (deflated 41%) 2022-03-17T22:12:57.8612260Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204438.xml (deflated 42%) 2022-03-17T22:12:57.8613015Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203132.xml (deflated 43%) 2022-03-17T22:12:57.8613757Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201513.xml (deflated 43%) 2022-03-17T22:12:57.8614488Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205155.xml (deflated 42%) 2022-03-17T22:12:57.8615232Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202704.xml (deflated 41%) 2022-03-17T22:12:57.8615987Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201551.xml (deflated 43%) 2022-03-17T22:12:57.8616712Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201646.xml (deflated 42%) 2022-03-17T22:12:57.8617465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205920.xml (deflated 42%) 2022-03-17T22:12:57.8618193Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204245.xml (deflated 42%) 2022-03-17T22:12:57.8618940Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204702.xml (deflated 42%) 2022-03-17T22:12:57.8619690Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203537.xml (deflated 43%) 2022-03-17T22:12:57.8620416Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203250.xml (deflated 41%) 2022-03-17T22:12:57.8621158Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201703.xml (deflated 41%) 2022-03-17T22:12:57.8621898Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204134.xml (deflated 40%) 2022-03-17T22:12:57.8622682Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202910.xml (deflated 41%) 2022-03-17T22:12:57.8623451Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201052.xml (deflated 48%) 2022-03-17T22:12:57.8624191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204331.xml (deflated 42%) 2022-03-17T22:12:57.8624924Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201725.xml (deflated 41%) 2022-03-17T22:12:57.8625660Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200915.xml (deflated 41%) 2022-03-17T22:12:57.8626417Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203504.xml (deflated 42%) 2022-03-17T22:12:57.8627145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201204.xml (deflated 40%) 2022-03-17T22:12:57.8627886Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204937.xml (deflated 40%) 2022-03-17T22:12:57.8628690Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205415.xml (deflated 41%) 2022-03-17T22:12:57.8629431Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201732.xml (deflated 41%) 2022-03-17T22:12:57.8630171Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210203.xml (deflated 43%) 2022-03-17T22:12:57.8630898Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203429.xml (deflated 41%) 2022-03-17T22:12:57.8631651Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210501.xml (deflated 44%) 2022-03-17T22:12:57.8632390Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200842.xml (deflated 40%) 2022-03-17T22:12:57.8633133Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201800.xml (deflated 42%) 2022-03-17T22:12:57.8633871Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201429.xml (deflated 43%) 2022-03-17T22:12:57.8634620Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201447.xml (deflated 42%) 2022-03-17T22:12:57.8635351Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201145.xml (deflated 40%) 2022-03-17T22:12:57.8636101Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204620.xml (deflated 42%) 2022-03-17T22:12:57.8636844Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210325.xml (deflated 42%) 2022-03-17T22:12:57.8637579Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204146.xml (deflated 40%) 2022-03-17T22:12:57.8638317Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203409.xml (deflated 41%) 2022-03-17T22:12:57.8639057Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210046.xml (deflated 42%) 2022-03-17T22:12:57.8639790Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210342.xml (deflated 41%) 2022-03-17T22:12:57.8640535Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203130.xml (deflated 43%) 2022-03-17T22:12:57.8641334Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210239.xml (deflated 41%) 2022-03-17T22:12:57.8642142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203412.xml (deflated 41%) 2022-03-17T22:12:57.8642883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201253.xml (deflated 42%) 2022-03-17T22:12:57.8643612Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204109.xml (deflated 42%) 2022-03-17T22:12:57.8644357Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201523.xml (deflated 43%) 2022-03-17T22:12:57.8645099Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203148.xml (deflated 42%) 2022-03-17T22:12:57.8645828Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210514.xml (deflated 43%) 2022-03-17T22:12:57.8646650Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201233.xml (deflated 41%) 2022-03-17T22:12:57.8647401Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201436.xml (deflated 43%) 2022-03-17T22:12:57.8648135Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203239.xml (deflated 42%) 2022-03-17T22:12:57.8648872Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204519.xml (deflated 41%) 2022-03-17T22:12:57.8649611Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201644.xml (deflated 43%) 2022-03-17T22:12:57.8650356Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205829.xml (deflated 41%) 2022-03-17T22:12:57.8651100Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202146.xml (deflated 42%) 2022-03-17T22:12:57.8651836Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201342.xml (deflated 41%) 2022-03-17T22:12:57.8652562Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202330.xml (deflated 42%) 2022-03-17T22:12:57.8653297Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205914.xml (deflated 40%) 2022-03-17T22:12:57.8654036Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204154.xml (deflated 40%) 2022-03-17T22:12:57.8654779Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203448.xml (deflated 41%) 2022-03-17T22:12:57.8655518Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204404.xml (deflated 43%) 2022-03-17T22:12:57.8656251Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210103.xml (deflated 42%) 2022-03-17T22:12:57.8656996Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210448.xml (deflated 41%) 2022-03-17T22:12:57.8657737Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205656.xml (deflated 41%) 2022-03-17T22:12:57.8658462Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204617.xml (deflated 43%) 2022-03-17T22:12:57.8659262Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205014.xml (deflated 41%) 2022-03-17T22:12:57.8660006Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202105.xml (deflated 41%) 2022-03-17T22:12:57.8660759Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202437.xml (deflated 41%) 2022-03-17T22:12:57.8661494Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204448.xml (deflated 41%) 2022-03-17T22:12:57.8662248Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210330.xml (deflated 41%) 2022-03-17T22:12:57.8662975Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210425.xml (deflated 41%) 2022-03-17T22:12:57.8663713Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201352.xml (deflated 42%) 2022-03-17T22:12:57.8664466Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203222.xml (deflated 41%) 2022-03-17T22:12:57.8665281Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205031.xml (deflated 41%) 2022-03-17T22:12:57.8666019Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210359.xml (deflated 42%) 2022-03-17T22:12:57.8666747Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204319.xml (deflated 43%) 2022-03-17T22:12:57.8667499Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201948.xml (deflated 42%) 2022-03-17T22:12:57.8668238Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204645.xml (deflated 41%) 2022-03-17T22:12:57.8668972Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203914.xml (deflated 40%) 2022-03-17T22:12:57.8669721Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202230.xml (deflated 42%) 2022-03-17T22:12:57.8670473Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201438.xml (deflated 43%) 2022-03-17T22:12:57.8671196Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201637.xml (deflated 42%) 2022-03-17T22:12:57.8671936Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210506.xml (deflated 43%) 2022-03-17T22:12:57.8672678Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204918.xml (deflated 41%) 2022-03-17T22:12:57.8673415Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204626.xml (deflated 43%) 2022-03-17T22:12:57.8674175Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205106.xml (deflated 41%) 2022-03-17T22:12:57.8674917Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200945.xml (deflated 41%) 2022-03-17T22:12:57.8675651Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201354.xml (deflated 42%) 2022-03-17T22:12:57.8676399Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205622.xml (deflated 41%) 2022-03-17T22:12:57.8677128Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210600.xml (deflated 42%) 2022-03-17T22:12:57.8677936Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204714.xml (deflated 41%) 2022-03-17T22:12:57.8678697Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203712.xml (deflated 42%) 2022-03-17T22:12:57.8679423Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201459.xml (deflated 41%) 2022-03-17T22:12:57.8680158Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203257.xml (deflated 42%) 2022-03-17T22:12:57.8680909Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205743.xml (deflated 40%) 2022-03-17T22:12:57.8681638Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204237.xml (deflated 42%) 2022-03-17T22:12:57.8682522Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210439.xml (deflated 44%) 2022-03-17T22:12:57.8683270Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204227.xml (deflated 41%) 2022-03-17T22:12:57.8684077Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203558.xml (deflated 41%) 2022-03-17T22:12:57.8684827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202153.xml (deflated 42%) 2022-03-17T22:12:57.8685560Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203934.xml (deflated 41%) 2022-03-17T22:12:57.8686290Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204359.xml (deflated 43%) 2022-03-17T22:12:57.8687034Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201609.xml (deflated 41%) 2022-03-17T22:12:57.8687768Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203551.xml (deflated 41%) 2022-03-17T22:12:57.8688512Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200835.xml (deflated 40%) 2022-03-17T22:12:57.8689261Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204550.xml (deflated 41%) 2022-03-17T22:12:57.8689985Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201258.xml (deflated 42%) 2022-03-17T22:12:57.8690721Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203832.xml (deflated 39%) 2022-03-17T22:12:57.8691458Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203543.xml (deflated 41%) 2022-03-17T22:12:57.8692191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205639.xml (deflated 41%) 2022-03-17T22:12:57.8692940Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202255.xml (deflated 41%) 2022-03-17T22:12:57.8693677Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204810.xml (deflated 41%) 2022-03-17T22:12:57.8694403Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203021.xml (deflated 40%) 2022-03-17T22:12:57.8695145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201808.xml (deflated 41%) 2022-03-17T22:12:57.8695881Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210142.xml (deflated 42%) 2022-03-17T22:12:57.8696673Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201015.xml (deflated 42%) 2022-03-17T22:12:57.8697437Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201553.xml (deflated 41%) 2022-03-17T22:12:57.8698175Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210517.xml (deflated 41%) 2022-03-17T22:12:57.8698901Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203302.xml (deflated 41%) 2022-03-17T22:12:57.8699646Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200752.xml (deflated 42%) 2022-03-17T22:12:57.8700371Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201110.xml (deflated 46%) 2022-03-17T22:12:57.8701116Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203320.xml (deflated 41%) 2022-03-17T22:12:57.8701927Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205632.xml (deflated 41%) 2022-03-17T22:12:57.8702650Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203529.xml (deflated 44%) 2022-03-17T22:12:57.8703407Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204637.xml (deflated 43%) 2022-03-17T22:12:57.8704143Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201954.xml (deflated 41%) 2022-03-17T22:12:57.8704868Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204440.xml (deflated 41%) 2022-03-17T22:12:57.8705616Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203403.xml (deflated 42%) 2022-03-17T22:12:57.8706369Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201137.xml (deflated 44%) 2022-03-17T22:12:57.8707094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201803.xml (deflated 42%) 2022-03-17T22:12:57.8707844Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204117.xml (deflated 44%) 2022-03-17T22:12:57.8708583Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210034.xml (deflated 43%) 2022-03-17T22:12:57.8709311Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203927.xml (deflated 41%) 2022-03-17T22:12:57.8710074Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203501.xml (deflated 42%) 2022-03-17T22:12:57.8710813Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203028.xml (deflated 40%) 2022-03-17T22:12:57.8711554Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204708.xml (deflated 41%) 2022-03-17T22:12:57.8712304Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201534.xml (deflated 42%) 2022-03-17T22:12:57.8713028Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204924.xml (deflated 41%) 2022-03-17T22:12:57.8713775Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202556.xml (deflated 41%) 2022-03-17T22:12:57.8714572Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204333.xml (deflated 42%) 2022-03-17T22:12:57.8715315Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205959.xml (deflated 41%) 2022-03-17T22:12:57.8716056Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203706.xml (deflated 42%) 2022-03-17T22:12:57.8716798Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210536.xml (deflated 40%) 2022-03-17T22:12:57.8717530Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203242.xml (deflated 42%) 2022-03-17T22:12:57.8718266Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210316.xml (deflated 42%) 2022-03-17T22:12:57.8719017Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203818.xml (deflated 40%) 2022-03-17T22:12:57.8719746Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201918.xml (deflated 41%) 2022-03-17T22:12:57.8720566Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210401.xml (deflated 42%) 2022-03-17T22:12:57.8721308Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201152.xml (deflated 40%) 2022-03-17T22:12:57.8722110Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201706.xml (deflated 41%) 2022-03-17T22:12:57.8722849Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202916.xml (deflated 41%) 2022-03-17T22:12:57.8723587Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204112.xml (deflated 44%) 2022-03-17T22:12:57.8724316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203441.xml (deflated 40%) 2022-03-17T22:12:57.8725059Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201311.xml (deflated 42%) 2022-03-17T22:12:57.8725785Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203018.xml (deflated 42%) 2022-03-17T22:12:57.8726534Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201417.xml (deflated 43%) 2022-03-17T22:12:57.8727272Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201600.xml (deflated 43%) 2022-03-17T22:12:57.8727999Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204954.xml (deflated 40%) 2022-03-17T22:12:57.8728740Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204306.xml (deflated 42%) 2022-03-17T22:12:57.8729490Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210052.xml (deflated 42%) 2022-03-17T22:12:57.8730217Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205909.xml (deflated 42%) 2022-03-17T22:12:57.8730960Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204736.xml (deflated 41%) 2022-03-17T22:12:57.8731708Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210404.xml (deflated 42%) 2022-03-17T22:12:57.8732431Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203035.xml (deflated 41%) 2022-03-17T22:12:57.8733238Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210423.xml (deflated 42%) 2022-03-17T22:12:57.8733988Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203136.xml (deflated 43%) 2022-03-17T22:12:57.8734711Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203208.xml (deflated 41%) 2022-03-17T22:12:57.8735460Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205506.xml (deflated 41%) 2022-03-17T22:12:57.8736203Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205203.xml (deflated 42%) 2022-03-17T22:12:57.8736928Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203649.xml (deflated 41%) 2022-03-17T22:12:57.8737669Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205141.xml (deflated 42%) 2022-03-17T22:12:57.8738403Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202225.xml (deflated 42%) 2022-03-17T22:12:57.8739212Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200849.xml (deflated 39%) 2022-03-17T22:12:57.8739956Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201327.xml (deflated 42%) 2022-03-17T22:12:57.8740687Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204240.xml (deflated 42%) 2022-03-17T22:12:57.8741438Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204120.xml (deflated 44%) 2022-03-17T22:12:57.8742186Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201655.xml (deflated 43%) 2022-03-17T22:12:57.8742917Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203436.xml (deflated 42%) 2022-03-17T22:12:57.8743661Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205311.xml (deflated 41%) 2022-03-17T22:12:57.8744404Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203003.xml (deflated 42%) 2022-03-17T22:12:57.8745142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204733.xml (deflated 42%) 2022-03-17T22:12:57.8745880Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210037.xml (deflated 43%) 2022-03-17T22:12:57.8746618Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204648.xml (deflated 41%) 2022-03-17T22:12:57.8747347Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204743.xml (deflated 41%) 2022-03-17T22:12:57.8748088Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202213.xml (deflated 42%) 2022-03-17T22:12:57.8748817Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205218.xml (deflated 41%) 2022-03-17T22:12:57.8749567Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201225.xml (deflated 40%) 2022-03-17T22:12:57.8750303Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200951.xml (deflated 41%) 2022-03-17T22:12:57.8751033Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205341.xml (deflated 41%) 2022-03-17T22:12:57.8751833Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204610.xml (deflated 41%) 2022-03-17T22:12:57.8752611Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205902.xml (deflated 40%) 2022-03-17T22:12:57.8753344Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210441.xml (deflated 41%) 2022-03-17T22:12:57.8754085Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201635.xml (deflated 43%) 2022-03-17T22:12:57.8754827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201129.xml (deflated 44%) 2022-03-17T22:12:57.8755555Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204643.xml (deflated 41%) 2022-03-17T22:12:57.8756313Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210433.xml (deflated 43%) 2022-03-17T22:12:57.8757113Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201616.xml (deflated 42%) 2022-03-17T22:12:57.8757855Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210306.xml (deflated 42%) 2022-03-17T22:12:57.8758595Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210328.xml (deflated 42%) 2022-03-17T22:12:57.8759345Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203127.xml (deflated 43%) 2022-03-17T22:12:57.8760071Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202219.xml (deflated 42%) 2022-03-17T22:12:57.8760810Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204308.xml (deflated 42%) 2022-03-17T22:12:57.8761540Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317200736.xml (deflated 42%) 2022-03-17T22:12:57.8762414Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210029.xml (deflated 43%) 2022-03-17T22:12:57.8763169Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202222.xml (deflated 42%) 2022-03-17T22:12:57.8763897Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205147.xml (deflated 42%) 2022-03-17T22:12:57.8764634Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201303.xml (deflated 42%) 2022-03-17T22:12:57.8765374Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203008.xml (deflated 41%) 2022-03-17T22:12:57.8766106Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201243.xml (deflated 41%) 2022-03-17T22:12:57.8766863Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205123.xml (deflated 42%) 2022-03-17T22:12:57.8767598Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201506.xml (deflated 41%) 2022-03-17T22:12:57.8768326Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205911.xml (deflated 41%) 2022-03-17T22:12:57.8769063Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204255.xml (deflated 42%) 2022-03-17T22:12:57.8769815Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204628.xml (deflated 42%) 2022-03-17T22:12:57.8770608Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202205.xml (deflated 42%) 2022-03-17T22:12:57.8771366Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201357.xml (deflated 41%) 2022-03-17T22:12:57.8772118Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201454.xml (deflated 43%) 2022-03-17T22:12:57.8772853Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210413.xml (deflated 41%) 2022-03-17T22:12:57.8773587Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204429.xml (deflated 43%) 2022-03-17T22:12:57.8774314Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204115.xml (deflated 44%) 2022-03-17T22:12:57.8775060Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210303.xml (deflated 42%) 2022-03-17T22:12:57.8775860Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210511.xml (deflated 43%) 2022-03-17T22:12:57.8776566Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317221206.xml (deflated 40%) 2022-03-17T22:12:57.8777229Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221242.xml (deflated 39%) 2022-03-17T22:12:57.8777861Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221233.xml (deflated 38%) 2022-03-17T22:12:57.8778476Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221224.xml (deflated 39%) 2022-03-17T22:12:57.8779101Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221247.xml (deflated 39%) 2022-03-17T22:12:57.8779775Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317221159.xml (deflated 40%) 2022-03-17T22:12:57.8780438Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317221218.xml (deflated 40%) 2022-03-17T22:12:57.8781098Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PythonStoreTest-20220317221209.xml (deflated 40%) 2022-03-17T22:12:57.8781720Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221239.xml (deflated 39%) 2022-03-17T22:12:57.8782380Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317221202.xml (deflated 40%) 2022-03-17T22:12:57.8783055Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317221156.xml (deflated 40%) 2022-03-17T22:12:57.8783706Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousEnvTest-20220317221212.xml (deflated 40%) 2022-03-17T22:12:57.8784373Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317221215.xml (deflated 40%) 2022-03-17T22:12:57.8785018Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221236.xml (deflated 38%) 2022-03-17T22:12:57.8785635Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317221153.xml (deflated 40%) 2022-03-17T22:12:57.8786289Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousTest-20220317221221.xml (deflated 39%) 2022-03-17T22:12:57.8786907Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317221150.xml (deflated 40%) 2022-03-17T22:12:57.8787532Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221230.xml (deflated 39%) 2022-03-17T22:12:57.8788153Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317221227.xml (deflated 39%) 2022-03-17T22:12:57.8788910Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211350.xml (deflated 41%) 2022-03-17T22:12:57.8789677Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220228.xml (deflated 41%) 2022-03-17T22:12:57.8790425Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220331.xml (deflated 41%) 2022-03-17T22:12:57.8791155Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211459.xml (deflated 43%) 2022-03-17T22:12:57.8791896Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215011.xml (deflated 41%) 2022-03-17T22:12:57.8792643Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214701.xml (deflated 41%) 2022-03-17T22:12:57.8793388Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211218.xml (deflated 40%) 2022-03-17T22:12:57.8794190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210614.xml (deflated 41%) 2022-03-17T22:12:57.8794916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212155.xml (deflated 43%) 2022-03-17T22:12:57.8795663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214147.xml (deflated 42%) 2022-03-17T22:12:57.8796414Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220532.xml (deflated 42%) 2022-03-17T22:12:57.8797141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214316.xml (deflated 43%) 2022-03-17T22:12:57.8797888Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214946.xml (deflated 41%) 2022-03-17T22:12:57.8798636Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214353.xml (deflated 40%) 2022-03-17T22:12:57.8799361Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214627.xml (deflated 41%) 2022-03-17T22:12:57.8800109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212931.xml (deflated 41%) 2022-03-17T22:12:57.8800847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212844.xml (deflated 41%) 2022-03-17T22:12:57.8801575Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213854.xml (deflated 42%) 2022-03-17T22:12:57.8802493Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210830.xml (deflated 42%) 2022-03-17T22:12:57.8803239Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220251.xml (deflated 41%) 2022-03-17T22:12:57.8803990Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220559.xml (deflated 41%) 2022-03-17T22:12:57.8804729Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211507.xml (deflated 43%) 2022-03-17T22:12:57.8805476Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220940.xml (deflated 41%) 2022-03-17T22:12:57.8806200Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214854.xml (deflated 42%) 2022-03-17T22:12:57.8807012Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210621.xml (deflated 42%) 2022-03-17T22:12:57.8807757Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213257.xml (deflated 42%) 2022-03-17T22:12:57.8808502Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213503.xml (deflated 42%) 2022-03-17T22:12:57.8809242Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214406.xml (deflated 42%) 2022-03-17T22:12:57.8809972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211140.xml (deflated 41%) 2022-03-17T22:12:57.8810719Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211630.xml (deflated 40%) 2022-03-17T22:12:57.8811495Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214016.xml (deflated 40%) 2022-03-17T22:12:57.8812221Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213313.xml (deflated 42%) 2022-03-17T22:12:57.8813037Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215217.xml (deflated 43%) 2022-03-17T22:12:57.8813783Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213509.xml (deflated 42%) 2022-03-17T22:12:57.8814506Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215114.xml (deflated 41%) 2022-03-17T22:12:57.8815248Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213703.xml (deflated 41%) 2022-03-17T22:12:57.8815993Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214721.xml (deflated 42%) 2022-03-17T22:12:57.8816731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213918.xml (deflated 41%) 2022-03-17T22:12:57.8817472Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220921.xml (deflated 41%) 2022-03-17T22:12:57.8818210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212629.xml (deflated 41%) 2022-03-17T22:12:57.8818944Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213602.xml (deflated 41%) 2022-03-17T22:12:57.8819685Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220811.xml (deflated 43%) 2022-03-17T22:12:57.8820411Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215002.xml (deflated 42%) 2022-03-17T22:12:57.8821154Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213149.xml (deflated 41%) 2022-03-17T22:12:57.8821894Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214952.xml (deflated 41%) 2022-03-17T22:12:57.8822627Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212345.xml (deflated 41%) 2022-03-17T22:12:57.8823366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213801.xml (deflated 41%) 2022-03-17T22:12:57.8824105Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220746.xml (deflated 42%) 2022-03-17T22:12:57.8824832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214811.xml (deflated 42%) 2022-03-17T22:12:57.8825644Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211515.xml (deflated 43%) 2022-03-17T22:12:57.8826396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211953.xml (deflated 41%) 2022-03-17T22:12:57.8827120Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220018.xml (deflated 41%) 2022-03-17T22:12:57.8827851Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211032.xml (deflated 43%) 2022-03-17T22:12:57.8828586Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220344.xml (deflated 41%) 2022-03-17T22:12:57.8829314Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214601.xml (deflated 41%) 2022-03-17T22:12:57.8830634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211647.xml (deflated 41%) 2022-03-17T22:12:57.8831456Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211244.xml (deflated 42%) 2022-03-17T22:12:57.8832180Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213034.xml (deflated 42%) 2022-03-17T22:12:57.8832913Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212408.xml (deflated 41%) 2022-03-17T22:12:57.8833643Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212512.xml (deflated 41%) 2022-03-17T22:12:57.8834387Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215005.xml (deflated 41%) 2022-03-17T22:12:57.8835131Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220452.xml (deflated 42%) 2022-03-17T22:12:57.8835856Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212754.xml (deflated 41%) 2022-03-17T22:12:57.8836604Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215439.xml (deflated 43%) 2022-03-17T22:12:57.8837366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214432.xml (deflated 41%) 2022-03-17T22:12:57.8838211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211257.xml (deflated 42%) 2022-03-17T22:12:57.8838964Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213310.xml (deflated 42%) 2022-03-17T22:12:57.8839705Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211436.xml (deflated 42%) 2022-03-17T22:12:57.8840439Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220827.xml (deflated 42%) 2022-03-17T22:12:57.8841189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214724.xml (deflated 43%) 2022-03-17T22:12:57.8842084Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213143.xml (deflated 41%) 2022-03-17T22:12:57.8842816Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211509.xml (deflated 43%) 2022-03-17T22:12:57.8843555Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220610.xml (deflated 41%) 2022-03-17T22:12:57.8844282Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214940.xml (deflated 40%) 2022-03-17T22:12:57.8845114Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213550.xml (deflated 42%) 2022-03-17T22:12:57.8845865Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211523.xml (deflated 42%) 2022-03-17T22:12:57.8846589Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221108.xml (deflated 42%) 2022-03-17T22:12:57.8847324Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220350.xml (deflated 40%) 2022-03-17T22:12:57.8848061Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214522.xml (deflated 40%) 2022-03-17T22:12:57.8848791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213300.xml (deflated 43%) 2022-03-17T22:12:57.8849546Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214646.xml (deflated 41%) 2022-03-17T22:12:57.8850346Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211439.xml (deflated 42%) 2022-03-17T22:12:57.8851077Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212737.xml (deflated 41%) 2022-03-17T22:12:57.8851815Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213106.xml (deflated 41%) 2022-03-17T22:12:57.8852552Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215452.xml (deflated 43%) 2022-03-17T22:12:57.8853278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211916.xml (deflated 41%) 2022-03-17T22:12:57.8854020Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214719.xml (deflated 42%) 2022-03-17T22:12:57.8854753Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214814.xml (deflated 43%) 2022-03-17T22:12:57.8855501Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211757.xml (deflated 42%) 2022-03-17T22:12:57.8856241Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213005.xml (deflated 41%) 2022-03-17T22:12:57.8856970Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215337.xml (deflated 41%) 2022-03-17T22:12:57.8857711Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212304.xml (deflated 43%) 2022-03-17T22:12:57.8858454Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211134.xml (deflated 42%) 2022-03-17T22:12:57.8859181Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212401.xml (deflated 40%) 2022-03-17T22:12:57.8859919Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211704.xml (deflated 41%) 2022-03-17T22:12:57.8860663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213544.xml (deflated 40%) 2022-03-17T22:12:57.8861387Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211212.xml (deflated 41%) 2022-03-17T22:12:57.8862129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215433.xml (deflated 42%) 2022-03-17T22:12:57.8862921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212239.xml (deflated 42%) 2022-03-17T22:12:57.8863654Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210942.xml (deflated 40%) 2022-03-17T22:12:57.8864397Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214022.xml (deflated 40%) 2022-03-17T22:12:57.8865138Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213754.xml (deflated 41%) 2022-03-17T22:12:57.8865865Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211431.xml (deflated 43%) 2022-03-17T22:12:57.8866601Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214759.xml (deflated 42%) 2022-03-17T22:12:57.8867324Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220312.xml (deflated 43%) 2022-03-17T22:12:57.8868070Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212703.xml (deflated 41%) 2022-03-17T22:12:57.8868867Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212208.xml (deflated 43%) 2022-03-17T22:12:57.8869591Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212948.xml (deflated 41%) 2022-03-17T22:12:57.8870330Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214753.xml (deflated 43%) 2022-03-17T22:12:57.8871069Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220627.xml (deflated 41%) 2022-03-17T22:12:57.8871798Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211323.xml (deflated 41%) 2022-03-17T22:12:57.8872543Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212227.xml (deflated 42%) 2022-03-17T22:12:57.8873283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220955.xml (deflated 41%) 2022-03-17T22:12:57.8874011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212034.xml (deflated 41%) 2022-03-17T22:12:57.8874753Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214412.xml (deflated 42%) 2022-03-17T22:12:57.8875493Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214425.xml (deflated 41%) 2022-03-17T22:12:57.8876219Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211336.xml (deflated 41%) 2022-03-17T22:12:57.8876963Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211049.xml (deflated 41%) 2022-03-17T22:12:57.8877690Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220205.xml (deflated 41%) 2022-03-17T22:12:57.8878427Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220832.xml (deflated 41%) 2022-03-17T22:12:57.8879167Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220318.xml (deflated 41%) 2022-03-17T22:12:57.8879894Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220708.xml (deflated 41%) 2022-03-17T22:12:57.8880632Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210836.xml (deflated 41%) 2022-03-17T22:12:57.8881439Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213718.xml (deflated 42%) 2022-03-17T22:12:57.8882251Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213333.xml (deflated 41%) 2022-03-17T22:12:57.8882995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213228.xml (deflated 41%) 2022-03-17T22:12:57.8883736Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210703.xml (deflated 41%) 2022-03-17T22:12:57.8884462Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212149.xml (deflated 42%) 2022-03-17T22:12:57.8885202Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220733.xml (deflated 42%) 2022-03-17T22:12:57.8885949Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214727.xml (deflated 42%) 2022-03-17T22:12:57.8886740Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214208.xml (deflated 46%) 2022-03-17T22:12:57.8887478Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213742.xml (deflated 41%) 2022-03-17T22:12:57.8888210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212017.xml (deflated 41%) 2022-03-17T22:12:57.8888948Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214156.xml (deflated 47%) 2022-03-17T22:12:57.8889684Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213847.xml (deflated 42%) 2022-03-17T22:12:57.8890418Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213027.xml (deflated 41%) 2022-03-17T22:12:57.8891162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210954.xml (deflated 40%) 2022-03-17T22:12:57.8891911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210935.xml (deflated 41%) 2022-03-17T22:12:57.8892644Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221039.xml (deflated 41%) 2022-03-17T22:12:57.8893393Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211238.xml (deflated 40%) 2022-03-17T22:12:57.8894131Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213721.xml (deflated 42%) 2022-03-17T22:12:57.8894856Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220535.xml (deflated 42%) 2022-03-17T22:12:57.8895595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211517.xml (deflated 43%) 2022-03-17T22:12:57.8896336Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210922.xml (deflated 46%) 2022-03-17T22:12:57.8897063Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211007.xml (deflated 41%) 2022-03-17T22:12:57.8897804Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213632.xml (deflated 42%) 2022-03-17T22:12:57.8898540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211530.xml (deflated 43%) 2022-03-17T22:12:57.8899265Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221026.xml (deflated 41%) 2022-03-17T22:12:57.8900061Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220605.xml (deflated 42%) 2022-03-17T22:12:57.8900800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214832.xml (deflated 41%) 2022-03-17T22:12:57.8901540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211624.xml (deflated 42%) 2022-03-17T22:12:57.8902278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212221.xml (deflated 42%) 2022-03-17T22:12:57.8903006Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211717.xml (deflated 41%) 2022-03-17T22:12:57.8903743Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215051.xml (deflated 42%) 2022-03-17T22:12:57.8904491Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210900.xml (deflated 42%) 2022-03-17T22:12:57.8905285Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210651.xml (deflated 41%) 2022-03-17T22:12:57.8906025Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213951.xml (deflated 41%) 2022-03-17T22:12:57.8906769Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214729.xml (deflated 43%) 2022-03-17T22:12:57.8907491Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214542.xml (deflated 42%) 2022-03-17T22:12:57.8908229Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213623.xml (deflated 41%) 2022-03-17T22:12:57.8908977Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221056.xml (deflated 40%) 2022-03-17T22:12:57.8909707Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215000.xml (deflated 42%) 2022-03-17T22:12:57.8910452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214234.xml (deflated 42%) 2022-03-17T22:12:57.8911184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215611.xml (deflated 41%) 2022-03-17T22:12:57.8911932Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213534.xml (deflated 43%) 2022-03-17T22:12:57.8912673Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210734.xml (deflated 39%) 2022-03-17T22:12:57.8913405Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213041.xml (deflated 40%) 2022-03-17T22:12:57.8914145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212720.xml (deflated 42%) 2022-03-17T22:12:57.8914884Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211403.xml (deflated 41%) 2022-03-17T22:12:57.8915613Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214714.xml (deflated 43%) 2022-03-17T22:12:57.8916357Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214949.xml (deflated 41%) 2022-03-17T22:12:57.8917098Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212529.xml (deflated 41%) 2022-03-17T22:12:57.8917882Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215529.xml (deflated 42%) 2022-03-17T22:12:57.8918625Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211512.xml (deflated 43%) 2022-03-17T22:12:57.8919369Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213902.xml (deflated 42%) 2022-03-17T22:12:57.8920093Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220537.xml (deflated 42%) 2022-03-17T22:12:57.8920838Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211712.xml (deflated 41%) 2022-03-17T22:12:57.8921566Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211457.xml (deflated 43%) 2022-03-17T22:12:57.8922395Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211251.xml (deflated 42%) 2022-03-17T22:12:57.8923141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213657.xml (deflated 41%) 2022-03-17T22:12:57.8923957Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215211.xml (deflated 41%) 2022-03-17T22:12:57.8924700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214140.xml (deflated 42%) 2022-03-17T22:12:57.8925441Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215505.xml (deflated 42%) 2022-03-17T22:12:57.8926164Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213349.xml (deflated 41%) 2022-03-17T22:12:57.8926906Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214419.xml (deflated 42%) 2022-03-17T22:12:57.8927651Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213047.xml (deflated 40%) 2022-03-17T22:12:57.8928380Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220257.xml (deflated 41%) 2022-03-17T22:12:57.8929120Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220011.xml (deflated 41%) 2022-03-17T22:12:57.8929860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213315.xml (deflated 40%) 2022-03-17T22:12:57.8930590Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213203.xml (deflated 41%) 2022-03-17T22:12:57.8931332Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214400.xml (deflated 42%) 2022-03-17T22:12:57.8932079Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220404.xml (deflated 41%) 2022-03-17T22:12:57.8932806Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211502.xml (deflated 43%) 2022-03-17T22:12:57.8933545Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211037.xml (deflated 40%) 2022-03-17T22:12:57.8934270Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211843.xml (deflated 40%) 2022-03-17T22:12:57.8935007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212215.xml (deflated 43%) 2022-03-17T22:12:57.8935749Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213222.xml (deflated 43%) 2022-03-17T22:12:57.8936542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214303.xml (deflated 40%) 2022-03-17T22:12:57.8937293Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214957.xml (deflated 41%) 2022-03-17T22:12:57.8938030Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211452.xml (deflated 43%) 2022-03-17T22:12:57.8938763Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214840.xml (deflated 41%) 2022-03-17T22:12:57.8939514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213831.xml (deflated 41%) 2022-03-17T22:12:57.8940252Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213450.xml (deflated 40%) 2022-03-17T22:12:57.8940987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214809.xml (deflated 42%) 2022-03-17T22:12:57.8941791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212011.xml (deflated 41%) 2022-03-17T22:12:57.8942535Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214554.xml (deflated 41%) 2022-03-17T22:12:57.8943283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215754.xml (deflated 41%) 2022-03-17T22:12:57.8944030Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211342.xml (deflated 41%) 2022-03-17T22:12:57.8944757Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211547.xml (deflated 43%) 2022-03-17T22:12:57.8945547Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210928.xml (deflated 41%) 2022-03-17T22:12:57.8946356Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211417.xml (deflated 41%) 2022-03-17T22:12:57.8947168Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211424.xml (deflated 41%) 2022-03-17T22:12:57.8947919Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214756.xml (deflated 43%) 2022-03-17T22:12:57.8948665Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214240.xml (deflated 41%) 2022-03-17T22:12:57.8949392Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221102.xml (deflated 41%) 2022-03-17T22:12:57.8950171Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210915.xml (deflated 48%) 2022-03-17T22:12:57.8950922Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211442.xml (deflated 42%) 2022-03-17T22:12:57.8951661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212730.xml (deflated 41%) 2022-03-17T22:12:57.8952406Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220727.xml (deflated 40%) 2022-03-17T22:12:57.8953147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212202.xml (deflated 43%) 2022-03-17T22:12:57.8953877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220306.xml (deflated 41%) 2022-03-17T22:12:57.8954617Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211709.xml (deflated 41%) 2022-03-17T22:12:57.8955432Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212455.xml (deflated 41%) 2022-03-17T22:12:57.8956173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211205.xml (deflated 41%) 2022-03-17T22:12:57.8956919Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214738.xml (deflated 42%) 2022-03-17T22:12:57.8957663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211122.xml (deflated 42%) 2022-03-17T22:12:57.8958390Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211449.xml (deflated 42%) 2022-03-17T22:12:57.8959135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211316.xml (deflated 41%) 2022-03-17T22:12:57.8959882Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211153.xml (deflated 41%) 2022-03-17T22:12:57.8960659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211659.xml (deflated 41%) 2022-03-17T22:12:57.8961410Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210624.xml (deflated 42%) 2022-03-17T22:12:57.8962285Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215320.xml (deflated 41%) 2022-03-17T22:12:57.8963031Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211043.xml (deflated 40%) 2022-03-17T22:12:57.8963774Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213736.xml (deflated 41%) 2022-03-17T22:12:57.8964512Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215708.xml (deflated 42%) 2022-03-17T22:12:57.8965259Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213528.xml (deflated 41%) 2022-03-17T22:12:57.8966009Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220540.xml (deflated 42%) 2022-03-17T22:12:57.8966734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211034.xml (deflated 43%) 2022-03-17T22:12:57.8967475Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213100.xml (deflated 40%) 2022-03-17T22:12:57.8968228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212918.xml (deflated 41%) 2022-03-17T22:12:57.8968956Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213621.xml (deflated 41%) 2022-03-17T22:12:57.8969699Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212546.xml (deflated 41%) 2022-03-17T22:12:57.8970439Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210631.xml (deflated 42%) 2022-03-17T22:12:57.8971163Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220958.xml (deflated 41%) 2022-03-17T22:12:57.8971903Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221013.xml (deflated 41%) 2022-03-17T22:12:57.8972628Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211520.xml (deflated 43%) 2022-03-17T22:12:57.8973558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220427.xml (deflated 41%) 2022-03-17T22:12:57.8974321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211528.xml (deflated 42%) 2022-03-17T22:12:57.8975050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213119.xml (deflated 40%) 2022-03-17T22:12:57.8975795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220550.xml (deflated 42%) 2022-03-17T22:12:57.8976538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211618.xml (deflated 42%) 2022-03-17T22:12:57.8977262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212924.xml (deflated 41%) 2022-03-17T22:12:57.8978003Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220142.xml (deflated 41%) 2022-03-17T22:12:57.8978748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211225.xml (deflated 40%) 2022-03-17T22:12:57.8979542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213748.xml (deflated 41%) 2022-03-17T22:12:57.8980279Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212328.xml (deflated 41%) 2022-03-17T22:12:57.8981015Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211055.xml (deflated 41%) 2022-03-17T22:12:57.8981740Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214347.xml (deflated 41%) 2022-03-17T22:12:57.8982486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214214.xml (deflated 41%) 2022-03-17T22:12:57.8983215Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212051.xml (deflated 41%) 2022-03-17T22:12:57.8983963Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211504.xml (deflated 43%) 2022-03-17T22:12:57.8984706Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214607.xml (deflated 41%) 2022-03-17T22:12:57.8985431Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220844.xml (deflated 41%) 2022-03-17T22:12:57.8986178Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213011.xml (deflated 41%) 2022-03-17T22:12:57.8986929Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214740.xml (deflated 43%) 2022-03-17T22:12:57.8987662Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214707.xml (deflated 40%) 2022-03-17T22:12:57.8988408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214509.xml (deflated 40%) 2022-03-17T22:12:57.8989147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214227.xml (deflated 41%) 2022-03-17T22:12:57.8989870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220526.xml (deflated 42%) 2022-03-17T22:12:57.8990611Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211409.xml (deflated 41%) 2022-03-17T22:12:57.8991349Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211016.xml (deflated 42%) 2022-03-17T22:12:57.8992147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211731.xml (deflated 40%) 2022-03-17T22:12:57.8992899Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214954.xml (deflated 41%) 2022-03-17T22:12:57.8993643Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214640.xml (deflated 41%) 2022-03-17T22:12:57.8994367Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215018.xml (deflated 41%) 2022-03-17T22:12:57.8995107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215421.xml (deflated 42%) 2022-03-17T22:12:57.8995840Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212258.xml (deflated 43%) 2022-03-17T22:12:57.8996588Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213608.xml (deflated 41%) 2022-03-17T22:12:57.8997388Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215904.xml (deflated 40%) 2022-03-17T22:12:57.8998113Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215541.xml (deflated 43%) 2022-03-17T22:12:57.8998853Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210816.xml (deflated 40%) 2022-03-17T22:12:57.8999596Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210853.xml (deflated 42%) 2022-03-17T22:12:57.9000321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212602.xml (deflated 41%) 2022-03-17T22:12:57.9001069Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210903.xml (deflated 46%) 2022-03-17T22:12:57.9001806Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211604.xml (deflated 40%) 2022-03-17T22:12:57.9002611Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214743.xml (deflated 43%) 2022-03-17T22:12:57.9003354Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210823.xml (deflated 43%) 2022-03-17T22:12:57.9004091Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213406.xml (deflated 41%) 2022-03-17T22:12:57.9004824Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220714.xml (deflated 41%) 2022-03-17T22:12:57.9005566Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214310.xml (deflated 42%) 2022-03-17T22:12:57.9006294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211128.xml (deflated 42%) 2022-03-17T22:12:57.9007045Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211102.xml (deflated 41%) 2022-03-17T22:12:57.9007787Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213327.xml (deflated 42%) 2022-03-17T22:12:57.9008511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211738.xml (deflated 42%) 2022-03-17T22:12:57.9009251Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220721.xml (deflated 41%) 2022-03-17T22:12:57.9009998Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212622.xml (deflated 41%) 2022-03-17T22:12:57.9010792Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213305.xml (deflated 43%) 2022-03-17T22:12:57.9011557Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215057.xml (deflated 41%) 2022-03-17T22:12:57.9012304Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211803.xml (deflated 42%) 2022-03-17T22:12:57.9013028Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215511.xml (deflated 42%) 2022-03-17T22:12:57.9013776Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214301.xml (deflated 41%) 2022-03-17T22:12:57.9014522Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215811.xml (deflated 41%) 2022-03-17T22:12:57.9015254Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220758.xml (deflated 41%) 2022-03-17T22:12:57.9016060Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210759.xml (deflated 41%) 2022-03-17T22:12:57.9016793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220004.xml (deflated 40%) 2022-03-17T22:12:57.9017540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220804.xml (deflated 41%) 2022-03-17T22:12:57.9018281Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214029.xml (deflated 40%) 2022-03-17T22:12:57.9019013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220902.xml (deflated 40%) 2022-03-17T22:12:57.9019761Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214634.xml (deflated 41%) 2022-03-17T22:12:57.9020511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215523.xml (deflated 42%) 2022-03-17T22:12:57.9021238Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220433.xml (deflated 41%) 2022-03-17T22:12:57.9021980Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214928.xml (deflated 42%) 2022-03-17T22:12:57.9022723Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211356.xml (deflated 41%) 2022-03-17T22:12:57.9023455Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213614.xml (deflated 41%) 2022-03-17T22:12:57.9024204Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215257.xml (deflated 41%) 2022-03-17T22:12:57.9024954Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215911.xml (deflated 41%) 2022-03-17T22:12:57.9025679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213126.xml (deflated 41%) 2022-03-17T22:12:57.9026418Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221000.xml (deflated 41%) 2022-03-17T22:12:57.9027159Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213236.xml (deflated 40%) 2022-03-17T22:12:57.9027885Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214319.xml (deflated 42%) 2022-03-17T22:12:57.9028676Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213626.xml (deflated 41%) 2022-03-17T22:12:57.9029417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214438.xml (deflated 41%) 2022-03-17T22:12:57.9030169Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211533.xml (deflated 41%) 2022-03-17T22:12:57.9030915Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212901.xml (deflated 41%) 2022-03-17T22:12:57.9031650Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220838.xml (deflated 41%) 2022-03-17T22:12:57.9032395Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220357.xml (deflated 40%) 2022-03-17T22:12:57.9033135Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214202.xml (deflated 48%) 2022-03-17T22:12:57.9033868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215928.xml (deflated 41%) 2022-03-17T22:12:57.9034666Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214123.xml (deflated 41%) 2022-03-17T22:12:57.9035409Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213710.xml (deflated 41%) 2022-03-17T22:12:57.9036139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220934.xml (deflated 41%) 2022-03-17T22:12:57.9036877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212252.xml (deflated 43%) 2022-03-17T22:12:57.9037620Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214529.xml (deflated 42%) 2022-03-17T22:12:57.9038349Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213537.xml (deflated 41%) 2022-03-17T22:12:57.9039095Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221032.xml (deflated 41%) 2022-03-17T22:12:57.9039821Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213824.xml (deflated 40%) 2022-03-17T22:12:57.9040568Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213456.xml (deflated 42%) 2022-03-17T22:12:57.9041302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213242.xml (deflated 40%) 2022-03-17T22:12:57.9042107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213209.xml (deflated 43%) 2022-03-17T22:12:57.9042858Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220513.xml (deflated 40%) 2022-03-17T22:12:57.9043605Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210721.xml (deflated 40%) 2022-03-17T22:12:57.9044329Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220458.xml (deflated 42%) 2022-03-17T22:12:57.9045079Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215031.xml (deflated 40%) 2022-03-17T22:12:57.9045822Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211025.xml (deflated 42%) 2022-03-17T22:12:57.9046548Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211018.xml (deflated 41%) 2022-03-17T22:12:57.9047357Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211329.xml (deflated 41%) 2022-03-17T22:12:57.9048108Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214457.xml (deflated 41%) 2022-03-17T22:12:57.9048831Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215427.xml (deflated 42%) 2022-03-17T22:12:57.9049588Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214322.xml (deflated 40%) 2022-03-17T22:12:57.9050316Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212137.xml (deflated 42%) 2022-03-17T22:12:57.9051082Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213944.xml (deflated 41%) 2022-03-17T22:12:57.9051825Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212246.xml (deflated 42%) 2022-03-17T22:12:57.9052628Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214341.xml (deflated 41%) 2022-03-17T22:12:57.9053366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215628.xml (deflated 40%) 2022-03-17T22:12:57.9054107Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212438.xml (deflated 42%) 2022-03-17T22:12:57.9054838Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220158.xml (deflated 41%) 2022-03-17T22:12:57.9055588Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215223.xml (deflated 41%) 2022-03-17T22:12:57.9056335Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215714.xml (deflated 43%) 2022-03-17T22:12:57.9057064Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213553.xml (deflated 42%) 2022-03-17T22:12:57.9057813Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220914.xml (deflated 41%) 2022-03-17T22:12:57.9058555Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212311.xml (deflated 41%) 2022-03-17T22:12:57.9059288Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211550.xml (deflated 41%) 2022-03-17T22:12:57.9060034Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211707.xml (deflated 42%) 2022-03-17T22:12:57.9060776Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214900.xml (deflated 42%) 2022-03-17T22:12:57.9061513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210909.xml (deflated 47%) 2022-03-17T22:12:57.9062258Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215548.xml (deflated 42%) 2022-03-17T22:12:57.9062981Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220545.xml (deflated 42%) 2022-03-17T22:12:57.9063728Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212619.xml (deflated 42%) 2022-03-17T22:12:57.9064471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215517.xml (deflated 42%) 2022-03-17T22:12:57.9065196Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212431.xml (deflated 43%) 2022-03-17T22:12:57.9066028Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221006.xml (deflated 41%) 2022-03-17T22:12:57.9066782Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213444.xml (deflated 41%) 2022-03-17T22:12:57.9067513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213651.xml (deflated 40%) 2022-03-17T22:12:57.9068253Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220001.xml (deflated 41%) 2022-03-17T22:12:57.9068992Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220548.xml (deflated 42%) 2022-03-17T22:12:57.9069727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220337.xml (deflated 40%) 2022-03-17T22:12:57.9070471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214246.xml (deflated 40%) 2022-03-17T22:12:57.9071262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215415.xml (deflated 42%) 2022-03-17T22:12:57.9071994Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210657.xml (deflated 41%) 2022-03-17T22:12:57.9072735Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214334.xml (deflated 41%) 2022-03-17T22:12:57.9073462Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211231.xml (deflated 41%) 2022-03-17T22:12:57.9074199Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220440.xml (deflated 41%) 2022-03-17T22:12:57.9074944Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212143.xml (deflated 42%) 2022-03-17T22:12:57.9075676Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220908.xml (deflated 42%) 2022-03-17T22:12:57.9076415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213645.xml (deflated 41%) 2022-03-17T22:12:57.9077157Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215025.xml (deflated 42%) 2022-03-17T22:12:57.9077889Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212723.xml (deflated 40%) 2022-03-17T22:12:57.9078629Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215458.xml (deflated 42%) 2022-03-17T22:12:57.9079370Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220034.xml (deflated 41%) 2022-03-17T22:12:57.9080105Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214735.xml (deflated 43%) 2022-03-17T22:12:57.9080847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220505.xml (deflated 41%) 2022-03-17T22:12:57.9081588Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214503.xml (deflated 40%) 2022-03-17T22:12:57.9082448Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214847.xml (deflated 40%) 2022-03-17T22:12:57.9083190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214745.xml (deflated 42%) 2022-03-17T22:12:57.9083995Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215845.xml (deflated 41%) 2022-03-17T22:12:57.9084753Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211115.xml (deflated 42%) 2022-03-17T22:12:57.9085492Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211434.xml (deflated 42%) 2022-03-17T22:12:57.9086215Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220553.xml (deflated 41%) 2022-03-17T22:12:57.9086956Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213155.xml (deflated 41%) 2022-03-17T22:12:57.9087700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211109.xml (deflated 41%) 2022-03-17T22:12:57.9088426Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220211.xml (deflated 41%) 2022-03-17T22:12:57.9089172Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215945.xml (deflated 41%) 2022-03-17T22:12:57.9089973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213912.xml (deflated 42%) 2022-03-17T22:12:57.9090701Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214117.xml (deflated 42%) 2022-03-17T22:12:57.9091443Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214716.xml (deflated 43%) 2022-03-17T22:12:57.9092188Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215240.xml (deflated 41%) 2022-03-17T22:12:57.9092916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214804.xml (deflated 42%) 2022-03-17T22:12:57.9093665Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213025.xml (deflated 43%) 2022-03-17T22:12:57.9094417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214823.xml (deflated 42%) 2022-03-17T22:12:57.9095146Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213437.xml (deflated 41%) 2022-03-17T22:12:57.9095886Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213113.xml (deflated 41%) 2022-03-17T22:12:57.9096613Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214328.xml (deflated 40%) 2022-03-17T22:12:57.9097358Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214906.xml (deflated 42%) 2022-03-17T22:12:57.9098104Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220125.xml (deflated 40%) 2022-03-17T22:12:57.9098833Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213430.xml (deflated 41%) 2022-03-17T22:12:57.9099574Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213249.xml (deflated 42%) 2022-03-17T22:12:57.9100312Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213724.xml (deflated 41%) 2022-03-17T22:12:57.9101042Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214816.xml (deflated 41%) 2022-03-17T22:12:57.9101797Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215535.xml (deflated 42%) 2022-03-17T22:12:57.9102595Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214548.xml (deflated 42%) 2022-03-17T22:12:57.9103339Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220421.xml (deflated 41%) 2022-03-17T22:12:57.9104079Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213522.xml (deflated 41%) 2022-03-17T22:12:57.9104823Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213321.xml (deflated 41%) 2022-03-17T22:12:57.9105546Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211859.xml (deflated 41%) 2022-03-17T22:12:57.9106291Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220244.xml (deflated 40%) 2022-03-17T22:12:57.9107027Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211001.xml (deflated 41%) 2022-03-17T22:12:57.9107767Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215303.xml (deflated 41%) 2022-03-17T22:12:57.9108568Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213639.xml (deflated 41%) 2022-03-17T22:12:57.9109297Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210948.xml (deflated 42%) 2022-03-17T22:12:57.9109725Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210741.xml (deflated 40%) 2022-03-17T22:12:57.9110154Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214252.xml (deflated 40%) 2022-03-17T22:12:57.9110586Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220325.xml (deflated 40%) 2022-03-17T22:12:57.9111019Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220952.xml (deflated 42%) 2022-03-17T22:12:57.9111452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210728.xml (deflated 40%) 2022-03-17T22:12:57.9111878Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215147.xml (deflated 41%) 2022-03-17T22:12:57.9112294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213716.xml (deflated 42%) 2022-03-17T22:12:57.9112722Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211013.xml (deflated 42%) 2022-03-17T22:12:57.9113149Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213555.xml (deflated 41%) 2022-03-17T22:12:57.9113582Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220644.xml (deflated 40%) 2022-03-17T22:12:57.9114013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220851.xml (deflated 41%) 2022-03-17T22:12:57.9114437Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215554.xml (deflated 40%) 2022-03-17T22:12:57.9114851Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220752.xml (deflated 42%) 2022-03-17T22:12:57.9115277Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212233.xml (deflated 42%) 2022-03-17T22:12:57.9115706Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213938.xml (deflated 41%) 2022-03-17T22:12:57.9116186Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215038.xml (deflated 40%) 2022-03-17T22:12:57.9116626Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214111.xml (deflated 44%) 2022-03-17T22:12:57.9117055Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215651.xml (deflated 41%) 2022-03-17T22:12:57.9117479Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214516.xml (deflated 41%) 2022-03-17T22:12:57.9117892Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220813.xml (deflated 41%) 2022-03-17T22:12:57.9118329Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214221.xml (deflated 41%) 2022-03-17T22:12:57.9118762Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213515.xml (deflated 42%) 2022-03-17T22:12:57.9119243Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213302.xml (deflated 42%) 2022-03-17T22:12:57.9119670Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214047.xml (deflated 41%) 2022-03-17T22:12:57.9120097Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213307.xml (deflated 42%) 2022-03-17T22:12:57.9120512Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220608.xml (deflated 43%) 2022-03-17T22:12:57.9120942Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215045.xml (deflated 42%) 2022-03-17T22:12:57.9121371Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215204.xml (deflated 41%) 2022-03-17T22:12:57.9121801Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220303.xml (deflated 42%) 2022-03-17T22:12:57.9122327Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215131.xml (deflated 41%) 2022-03-17T22:12:57.9122756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220857.xml (deflated 42%) 2022-03-17T22:12:57.9123184Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215828.xml (deflated 41%) 2022-03-17T22:12:57.9123599Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213413.xml (deflated 40%) 2022-03-17T22:12:57.9124033Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220829.xml (deflated 42%) 2022-03-17T22:12:57.9124466Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213909.xml (deflated 42%) 2022-03-17T22:12:57.9124898Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211715.xml (deflated 41%) 2022-03-17T22:12:57.9125326Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213215.xml (deflated 42%) 2022-03-17T22:12:57.9125752Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211653.xml (deflated 40%) 2022-03-17T22:12:57.9126182Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214732.xml (deflated 42%) 2022-03-17T22:12:57.9126659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211826.xml (deflated 41%) 2022-03-17T22:12:57.9127097Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214535.xml (deflated 42%) 2022-03-17T22:12:57.9127528Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211750.xml (deflated 41%) 2022-03-17T22:12:57.9127959Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211454.xml (deflated 42%) 2022-03-17T22:12:57.9128386Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214008.xml (deflated 40%) 2022-03-17T22:12:57.9128813Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213818.xml (deflated 41%) 2022-03-17T22:12:57.9129228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211924.xml (deflated 41%) 2022-03-17T22:12:57.9129658Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211702.xml (deflated 41%) 2022-03-17T22:12:57.9130142Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220927.xml (deflated 40%) 2022-03-17T22:12:57.9130571Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212113.xml (deflated 41%) 2022-03-17T22:12:57.9130999Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214451.xml (deflated 41%) 2022-03-17T22:12:57.9131428Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212646.xml (deflated 41%) 2022-03-17T22:12:57.9131855Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212425.xml (deflated 42%) 2022-03-17T22:12:57.9132276Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220651.xml (deflated 40%) 2022-03-17T22:12:57.9132713Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214748.xml (deflated 42%) 2022-03-17T22:12:57.9133143Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220520.xml (deflated 41%) 2022-03-17T22:12:57.9133571Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221125.xml (deflated 42%) 2022-03-17T22:12:57.9133999Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213330.xml (deflated 43%) 2022-03-17T22:12:57.9134432Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214445.xml (deflated 41%) 2022-03-17T22:12:57.9134862Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220542.xml (deflated 43%) 2022-03-17T22:12:57.9135278Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214830.xml (deflated 43%) 2022-03-17T22:12:57.9135703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211638.xml (deflated 42%) 2022-03-17T22:12:57.9136133Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215901.xml (deflated 43%) 2022-03-17T22:12:57.9136558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211724.xml (deflated 41%) 2022-03-17T22:12:57.9136985Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214913.xml (deflated 40%) 2022-03-17T22:12:57.9137460Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213018.xml (deflated 41%) 2022-03-17T22:12:57.9137883Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214621.xml (deflated 40%) 2022-03-17T22:12:57.9138307Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211444.xml (deflated 42%) 2022-03-17T22:12:57.9138734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215720.xml (deflated 42%) 2022-03-17T22:12:57.9139162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211809.xml (deflated 41%) 2022-03-17T22:12:57.9139586Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220946.xml (deflated 40%) 2022-03-17T22:12:57.9140021Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221131.xml (deflated 42%) 2022-03-17T22:12:57.9140454Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211612.xml (deflated 42%) 2022-03-17T22:12:57.9140922Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215644.xml (deflated 40%) 2022-03-17T22:12:57.9141361Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211641.xml (deflated 42%) 2022-03-17T22:12:57.9141791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214614.xml (deflated 40%) 2022-03-17T22:12:57.9142218Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214934.xml (deflated 40%) 2022-03-17T22:12:57.9142651Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220859.xml (deflated 41%) 2022-03-17T22:12:57.9143082Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211525.xml (deflated 42%) 2022-03-17T22:12:57.9143502Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214653.xml (deflated 41%) 2022-03-17T22:12:57.9143932Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213255.xml (deflated 42%) 2022-03-17T22:12:57.9144364Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220446.xml (deflated 43%) 2022-03-17T22:12:57.9144795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211744.xml (deflated 41%) 2022-03-17T22:12:57.9145221Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214104.xml (deflated 40%) 2022-03-17T22:12:57.9145654Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213054.xml (deflated 41%) 2022-03-17T22:12:57.9146084Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215358.xml (deflated 41%) 2022-03-17T22:12:57.9146498Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211936.xml (deflated 41%) 2022-03-17T22:12:57.9146927Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317221019.xml (deflated 41%) 2022-03-17T22:12:57.9147355Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211557.xml (deflated 41%) 2022-03-17T22:12:57.9147784Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214920.xml (deflated 41%) 2022-03-17T22:12:57.9148268Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212130.xml (deflated 43%) 2022-03-17T22:12:57.9148710Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211147.xml (deflated 41%) 2022-03-17T22:12:57.9149146Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211447.xml (deflated 43%) 2022-03-17T22:12:57.9149560Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211159.xml (deflated 41%) 2022-03-17T22:12:57.9149987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211310.xml (deflated 41%) 2022-03-17T22:12:57.9150414Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211541.xml (deflated 42%) 2022-03-17T22:12:57.9150847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213730.xml (deflated 41%) 2022-03-17T22:12:57.9151325Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220820.xml (deflated 41%) 2022-03-17T22:12:57.9151753Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211930.xml (deflated 43%) 2022-03-17T22:12:57.9152166Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214150.xml (deflated 46%) 2022-03-17T22:12:57.9152591Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215446.xml (deflated 42%) 2022-03-17T22:12:57.9153016Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214258.xml (deflated 42%) 2022-03-17T22:12:57.9153446Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214751.xml (deflated 43%) 2022-03-17T22:12:57.9153880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214806.xml (deflated 42%) 2022-03-17T22:12:57.9154307Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317215737.xml (deflated 41%) 2022-03-17T22:12:57.9154739Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317214801.xml (deflated 43%) 2022-03-17T22:12:57.9155150Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211303.xml (deflated 42%) 2022-03-17T22:12:57.9155581Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317220739.xml (deflated 43%) 2022-03-17T22:12:57.9155952Z adding: test/test-reports/dist-gloo/distributed.test_nccl/TEST-TestNCCLCUDA-20220317221143.xml (deflated 83%) 2022-03-17T22:12:57.9156352Z adding: test/test-reports/dist-gloo/distributed.test_launcher/TEST-TestDistributedLaunch-20220317221138.xml (deflated 46%) 2022-03-17T22:12:57.9156845Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20220317200706.xml (deflated 86%) 2022-03-17T22:12:57.9157263Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220317200706.xml (deflated 82%) 2022-03-17T22:12:57.9157753Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220317192702.xml (deflated 80%) 2022-03-17T22:12:57.9158208Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220317192702.xml (deflated 79%) 2022-03-17T22:12:57.9158727Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220317192702.xml (deflated 55%) 2022-03-17T22:12:57.9159163Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220317192702.xml (deflated 63%) 2022-03-17T22:12:57.9159631Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200651.xml (deflated 42%) 2022-03-17T22:12:57.9160100Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200644.xml (deflated 42%) 2022-03-17T22:12:57.9160558Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200547.xml (deflated 41%) 2022-03-17T22:12:57.9161007Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200554.xml (deflated 41%) 2022-03-17T22:12:57.9161480Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200616.xml (deflated 42%) 2022-03-17T22:12:57.9162087Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200658.xml (deflated 42%) 2022-03-17T22:12:57.9162646Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200637.xml (deflated 42%) 2022-03-17T22:12:57.9163109Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200609.xml (deflated 41%) 2022-03-17T22:12:57.9163572Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200623.xml (deflated 42%) 2022-03-17T22:12:57.9164037Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317200630.xml (deflated 43%) 2022-03-17T22:12:57.9164488Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317200602.xml (deflated 41%) 2022-03-17T22:12:57.9164945Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200426.xml (deflated 41%) 2022-03-17T22:12:57.9165419Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200502.xml (deflated 42%) 2022-03-17T22:12:57.9165883Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200448.xml (deflated 42%) 2022-03-17T22:12:57.9166391Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200400.xml (deflated 44%) 2022-03-17T22:12:57.9166899Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200344.xml (deflated 44%) 2022-03-17T22:12:57.9167365Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200537.xml (deflated 42%) 2022-03-17T22:12:57.9167867Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200347.xml (deflated 44%) 2022-03-17T22:12:57.9168334Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200530.xml (deflated 42%) 2022-03-17T22:12:57.9168786Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200433.xml (deflated 41%) 2022-03-17T22:12:57.9169255Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200455.xml (deflated 42%) 2022-03-17T22:12:57.9169722Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200516.xml (deflated 42%) 2022-03-17T22:12:57.9170233Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200441.xml (deflated 41%) 2022-03-17T22:12:57.9170703Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200419.xml (deflated 41%) 2022-03-17T22:12:57.9171155Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200523.xml (deflated 42%) 2022-03-17T22:12:57.9171613Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200509.xml (deflated 42%) 2022-03-17T22:12:57.9172068Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220317192736.xml (deflated 54%) 2022-03-17T22:12:57.9172531Z adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220317192756.xml (deflated 71%) 2022-03-17T22:12:57.9172961Z adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220317192659.xml (deflated 63%) 2022-03-17T22:12:57.9173456Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193717.xml (deflated 47%) 2022-03-17T22:12:57.9173897Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194112.xml (deflated 41%) 2022-03-17T22:12:57.9174324Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193901.xml (deflated 44%) 2022-03-17T22:12:57.9174762Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193839.xml (deflated 44%) 2022-03-17T22:12:57.9175181Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194510.xml (deflated 39%) 2022-03-17T22:12:57.9175603Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194503.xml (deflated 40%) 2022-03-17T22:12:57.9175982Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193430.xml (deflated 39%) 2022-03-17T22:12:57.9176439Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193914.xml (deflated 44%) 2022-03-17T22:12:57.9176844Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194543.xml (deflated 39%) 2022-03-17T22:12:57.9177263Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194231.xml (deflated 40%) 2022-03-17T22:12:57.9177629Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193503.xml (deflated 39%) 2022-03-17T22:12:57.9178052Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194336.xml (deflated 41%) 2022-03-17T22:12:57.9178421Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193508.xml (deflated 38%) 2022-03-17T22:12:57.9178874Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193926.xml (deflated 45%) 2022-03-17T22:12:57.9179286Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194238.xml (deflated 39%) 2022-03-17T22:12:57.9179659Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220317194659.xml (deflated 40%) 2022-03-17T22:12:57.9180075Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194414.xml (deflated 41%) 2022-03-17T22:12:57.9180495Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194330.xml (deflated 40%) 2022-03-17T22:12:57.9180992Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194134.xml (deflated 42%) 2022-03-17T22:12:57.9181449Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193822.xml (deflated 44%) 2022-03-17T22:12:57.9181897Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193627.xml (deflated 45%) 2022-03-17T22:12:57.9182270Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193443.xml (deflated 38%) 2022-03-17T22:12:57.9182675Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194603.xml (deflated 40%) 2022-03-17T22:12:57.9183052Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194645.xml (deflated 39%) 2022-03-17T22:12:57.9183480Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194524.xml (deflated 39%) 2022-03-17T22:12:57.9183964Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194609.xml (deflated 40%) 2022-03-17T22:12:57.9184390Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194355.xml (deflated 40%) 2022-03-17T22:12:57.9184834Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193932.xml (deflated 50%) 2022-03-17T22:12:57.9185235Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194417.xml (deflated 40%) 2022-03-17T22:12:57.9185615Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193436.xml (deflated 38%) 2022-03-17T22:12:57.9186031Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194444.xml (deflated 40%) 2022-03-17T22:12:57.9186458Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194550.xml (deflated 39%) 2022-03-17T22:12:57.9186867Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220317194656.xml (deflated 39%) 2022-03-17T22:12:57.9187305Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193806.xml (deflated 46%) 2022-03-17T22:12:57.9187727Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194153.xml (deflated 39%) 2022-03-17T22:12:57.9188157Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193749.xml (deflated 45%) 2022-03-17T22:12:57.9188575Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194407.xml (deflated 40%) 2022-03-17T22:12:57.9189028Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194031.xml (deflated 42%) 2022-03-17T22:12:57.9189456Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194624.xml (deflated 41%) 2022-03-17T22:12:57.9189905Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194014.xml (deflated 42%) 2022-03-17T22:12:57.9190316Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194518.xml (deflated 39%) 2022-03-17T22:12:57.9190743Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194100.xml (deflated 46%) 2022-03-17T22:12:57.9191163Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194245.xml (deflated 39%) 2022-03-17T22:12:57.9191654Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193951.xml (deflated 41%) 2022-03-17T22:12:57.9192104Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193554.xml (deflated 43%) 2022-03-17T22:12:57.9192547Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193520.xml (deflated 45%) 2022-03-17T22:12:57.9192930Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194640.xml (deflated 39%) 2022-03-17T22:12:57.9193307Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194651.xml (deflated 38%) 2022-03-17T22:12:57.9193709Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194212.xml (deflated 40%) 2022-03-17T22:12:57.9194126Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194316.xml (deflated 40%) 2022-03-17T22:12:57.9194552Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194430.xml (deflated 40%) 2022-03-17T22:12:57.9195025Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194423.xml (deflated 40%) 2022-03-17T22:12:57.9195475Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193610.xml (deflated 43%) 2022-03-17T22:12:57.9195886Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194310.xml (deflated 40%) 2022-03-17T22:12:57.9196253Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194648.xml (deflated 39%) 2022-03-17T22:12:57.9196641Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194643.xml (deflated 39%) 2022-03-17T22:12:57.9197099Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193908.xml (deflated 46%) 2022-03-17T22:12:57.9197550Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193945.xml (deflated 41%) 2022-03-17T22:12:57.9198000Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194055.xml (deflated 44%) 2022-03-17T22:12:57.9198370Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193514.xml (deflated 38%) 2022-03-17T22:12:57.9198817Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193700.xml (deflated 47%) 2022-03-17T22:12:57.9199218Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194303.xml (deflated 40%) 2022-03-17T22:12:57.9199634Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194206.xml (deflated 40%) 2022-03-17T22:12:57.9200047Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194159.xml (deflated 40%) 2022-03-17T22:12:57.9200501Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193920.xml (deflated 45%) 2022-03-17T22:12:57.9200919Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194437.xml (deflated 40%) 2022-03-17T22:12:57.9201361Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194042.xml (deflated 41%) 2022-03-17T22:12:57.9201806Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193537.xml (deflated 45%) 2022-03-17T22:12:57.9202315Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193939.xml (deflated 41%) 2022-03-17T22:12:57.9202836Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193855.xml (deflated 44%) 2022-03-17T22:12:57.9203279Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194634.xml (deflated 40%) 2022-03-17T22:12:57.9203704Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194626.xml (deflated 40%) 2022-03-17T22:12:57.9204115Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194450.xml (deflated 41%) 2022-03-17T22:12:57.9204533Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194556.xml (deflated 39%) 2022-03-17T22:12:57.9204934Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194348.xml (deflated 40%) 2022-03-17T22:12:57.9205357Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194146.xml (deflated 40%) 2022-03-17T22:12:57.9205866Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193957.xml (deflated 41%) 2022-03-17T22:12:57.9206289Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194224.xml (deflated 40%) 2022-03-17T22:12:57.9206698Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194323.xml (deflated 40%) 2022-03-17T22:12:57.9207067Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193456.xml (deflated 38%) 2022-03-17T22:12:57.9207487Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194615.xml (deflated 41%) 2022-03-17T22:12:57.9207915Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193733.xml (deflated 44%) 2022-03-17T22:12:57.9208330Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194140.xml (deflated 40%) 2022-03-17T22:12:57.9208757Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194536.xml (deflated 39%) 2022-03-17T22:12:57.9209167Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194618.xml (deflated 40%) 2022-03-17T22:12:57.9209605Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194128.xml (deflated 42%) 2022-03-17T22:12:57.9210019Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194456.xml (deflated 40%) 2022-03-17T22:12:57.9210454Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194036.xml (deflated 42%) 2022-03-17T22:12:57.9210847Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317194654.xml (deflated 39%) 2022-03-17T22:12:57.9211272Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194531.xml (deflated 41%) 2022-03-17T22:12:57.9211689Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194218.xml (deflated 41%) 2022-03-17T22:12:57.9212105Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194251.xml (deflated 40%) 2022-03-17T22:12:57.9212530Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194257.xml (deflated 40%) 2022-03-17T22:12:57.9212904Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317193449.xml (deflated 38%) 2022-03-17T22:12:57.9213359Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194401.xml (deflated 40%) 2022-03-17T22:12:57.9213815Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317193643.xml (deflated 46%) 2022-03-17T22:12:57.9214251Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194106.xml (deflated 41%) 2022-03-17T22:12:57.9214698Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317194049.xml (deflated 41%) 2022-03-17T22:12:57.9215117Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317194342.xml (deflated 40%) 2022-03-17T22:12:57.9215660Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220317192808.xml (deflated 73%) 2022-03-17T22:12:57.9216208Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220317192808.xml (deflated 90%) 2022-03-17T22:12:57.9216669Z adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220317192802.xml (deflated 55%) 2022-03-17T22:12:57.9217068Z adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220317192622.xml (deflated 79%) 2022-03-17T22:12:57.9217537Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193415.xml (deflated 41%) 2022-03-17T22:12:57.9218007Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193421.xml (deflated 41%) 2022-03-17T22:12:57.9218454Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193348.xml (deflated 41%) 2022-03-17T22:12:57.9218921Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193400.xml (deflated 41%) 2022-03-17T22:12:57.9219380Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317193407.xml (deflated 41%) 2022-03-17T22:12:57.9219822Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193354.xml (deflated 40%) 2022-03-17T22:12:57.9220269Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193351.xml (deflated 40%) 2022-03-17T22:12:57.9220651Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220317193343.xml (deflated 38%) 2022-03-17T22:12:57.9221096Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317193357.xml (deflated 42%) 2022-03-17T22:12:57.9221564Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220317192749.xml (deflated 71%) 2022-03-17T22:12:57.9222015Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220317192749.xml (deflated 69%) 2022-03-17T22:12:57.9222503Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220317192749.xml (deflated 66%) 2022-03-17T22:12:57.9222941Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195113.xml (deflated 46%) 2022-03-17T22:12:57.9223379Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195056.xml (deflated 43%) 2022-03-17T22:12:57.9223820Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200034.xml (deflated 42%) 2022-03-17T22:12:57.9224243Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194837.xml (deflated 38%) 2022-03-17T22:12:57.9224668Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200312.xml (deflated 39%) 2022-03-17T22:12:57.9225118Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194952.xml (deflated 41%) 2022-03-17T22:12:57.9225543Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195440.xml (deflated 41%) 2022-03-17T22:12:57.9225961Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200221.xml (deflated 39%) 2022-03-17T22:12:57.9226332Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194812.xml (deflated 38%) 2022-03-17T22:12:57.9226777Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195001.xml (deflated 42%) 2022-03-17T22:12:57.9227203Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200325.xml (deflated 39%) 2022-03-17T22:12:57.9227700Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195909.xml (deflated 43%) 2022-03-17T22:12:57.9228129Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195447.xml (deflated 42%) 2022-03-17T22:12:57.9228568Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200046.xml (deflated 41%) 2022-03-17T22:12:57.9229008Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195429.xml (deflated 41%) 2022-03-17T22:12:57.9229456Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195147.xml (deflated 47%) 2022-03-17T22:12:57.9229906Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195236.xml (deflated 45%) 2022-03-17T22:12:57.9230349Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194917.xml (deflated 42%) 2022-03-17T22:12:57.9230798Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195916.xml (deflated 42%) 2022-03-17T22:12:57.9231223Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195416.xml (deflated 42%) 2022-03-17T22:12:57.9231665Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195326.xml (deflated 44%) 2022-03-17T22:12:57.9232100Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195626.xml (deflated 41%) 2022-03-17T22:12:57.9232539Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195735.xml (deflated 42%) 2022-03-17T22:12:57.9232976Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194949.xml (deflated 41%) 2022-03-17T22:12:57.9233348Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194745.xml (deflated 39%) 2022-03-17T22:12:57.9233762Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200233.xml (deflated 39%) 2022-03-17T22:12:57.9234168Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200114.xml (deflated 41%) 2022-03-17T22:12:57.9234616Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195403.xml (deflated 44%) 2022-03-17T22:12:57.9235104Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195706.xml (deflated 42%) 2022-03-17T22:12:57.9235549Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194954.xml (deflated 41%) 2022-03-17T22:12:57.9235986Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195356.xml (deflated 42%) 2022-03-17T22:12:57.9236422Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195342.xml (deflated 42%) 2022-03-17T22:12:57.9236784Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194708.xml (deflated 39%) 2022-03-17T22:12:57.9237160Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194806.xml (deflated 38%) 2022-03-17T22:12:57.9237591Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200125.xml (deflated 41%) 2022-03-17T22:12:57.9238009Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200053.xml (deflated 41%) 2022-03-17T22:12:57.9238469Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220317200331.xml (deflated 40%) 2022-03-17T22:12:57.9238906Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195129.xml (deflated 46%) 2022-03-17T22:12:57.9239327Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200105.xml (deflated 42%) 2022-03-17T22:12:57.9239751Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195024.xml (deflated 45%) 2022-03-17T22:12:57.9240196Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200028.xml (deflated 42%) 2022-03-17T22:12:57.9240639Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195350.xml (deflated 42%) 2022-03-17T22:12:57.9241084Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195409.xml (deflated 44%) 2022-03-17T22:12:57.9241461Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194819.xml (deflated 38%) 2022-03-17T22:12:57.9241977Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195553.xml (deflated 41%) 2022-03-17T22:12:57.9242406Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195852.xml (deflated 42%) 2022-03-17T22:12:57.9242788Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194800.xml (deflated 38%) 2022-03-17T22:12:57.9243204Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200259.xml (deflated 39%) 2022-03-17T22:12:57.9243635Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200131.xml (deflated 43%) 2022-03-17T22:12:57.9244076Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195007.xml (deflated 45%) 2022-03-17T22:12:57.9244459Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220317200334.xml (deflated 40%) 2022-03-17T22:12:57.9244882Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200142.xml (deflated 38%) 2022-03-17T22:12:57.9245311Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195949.xml (deflated 42%) 2022-03-17T22:12:57.9245749Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220317200140.xml (deflated 42%) 2022-03-17T22:12:57.9246266Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195712.xml (deflated 41%) 2022-03-17T22:12:57.9246701Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200214.xml (deflated 39%) 2022-03-17T22:12:57.9247138Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195536.xml (deflated 41%) 2022-03-17T22:12:57.9247584Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194900.xml (deflated 41%) 2022-03-17T22:12:57.9247961Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194726.xml (deflated 38%) 2022-03-17T22:12:57.9248371Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200202.xml (deflated 40%) 2022-03-17T22:12:57.9248818Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194843.xml (deflated 41%) 2022-03-17T22:12:57.9249296Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200252.xml (deflated 39%) 2022-03-17T22:12:57.9249713Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200240.xml (deflated 39%) 2022-03-17T22:12:57.9250147Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195309.xml (deflated 44%) 2022-03-17T22:12:57.9250531Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194754.xml (deflated 37%) 2022-03-17T22:12:57.9250964Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195434.xml (deflated 41%) 2022-03-17T22:12:57.9251408Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195858.xml (deflated 42%) 2022-03-17T22:12:57.9251847Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195610.xml (deflated 41%) 2022-03-17T22:12:57.9252221Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194721.xml (deflated 39%) 2022-03-17T22:12:57.9252673Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195040.xml (deflated 43%) 2022-03-17T22:12:57.9253098Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200208.xml (deflated 39%) 2022-03-17T22:12:57.9253538Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195219.xml (deflated 45%) 2022-03-17T22:12:57.9253897Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194714.xml (deflated 39%) 2022-03-17T22:12:57.9254269Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194751.xml (deflated 40%) 2022-03-17T22:12:57.9254688Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200108.xml (deflated 42%) 2022-03-17T22:12:57.9255057Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194831.xml (deflated 38%) 2022-03-17T22:12:57.9255493Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195203.xml (deflated 47%) 2022-03-17T22:12:57.9255930Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200040.xml (deflated 41%) 2022-03-17T22:12:57.9256284Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194739.xml (deflated 40%) 2022-03-17T22:12:57.9256769Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195422.xml (deflated 41%) 2022-03-17T22:12:57.9257204Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200318.xml (deflated 39%) 2022-03-17T22:12:57.9257630Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200227.xml (deflated 39%) 2022-03-17T22:12:57.9258070Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195903.xml (deflated 41%) 2022-03-17T22:12:57.9258509Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195836.xml (deflated 42%) 2022-03-17T22:12:57.9258949Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195933.xml (deflated 42%) 2022-03-17T22:12:57.9259385Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200011.xml (deflated 45%) 2022-03-17T22:12:57.9259822Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195643.xml (deflated 42%) 2022-03-17T22:12:57.9260320Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195746.xml (deflated 42%) 2022-03-17T22:12:57.9260734Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200155.xml (deflated 39%) 2022-03-17T22:12:57.9261157Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200246.xml (deflated 39%) 2022-03-17T22:12:57.9261600Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195659.xml (deflated 41%) 2022-03-17T22:12:57.9262016Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200119.xml (deflated 41%) 2022-03-17T22:12:57.9262424Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200134.xml (deflated 42%) 2022-03-17T22:12:57.9262997Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195504.xml (deflated 41%) 2022-03-17T22:12:57.9264487Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200149.xml (deflated 39%) 2022-03-17T22:12:57.9264977Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194825.xml (deflated 39%) 2022-03-17T22:12:57.9265511Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195252.xml (deflated 46%) 2022-03-17T22:12:57.9266028Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195520.xml (deflated 41%) 2022-03-17T22:12:57.9266512Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200059.xml (deflated 41%) 2022-03-17T22:12:57.9266954Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317194733.xml (deflated 38%) 2022-03-17T22:12:57.9267472Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195728.xml (deflated 42%) 2022-03-17T22:12:57.9267987Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317194933.xml (deflated 41%) 2022-03-17T22:12:57.9268687Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195803.xml (deflated 42%) 2022-03-17T22:12:57.9269203Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195819.xml (deflated 41%) 2022-03-17T22:12:57.9269831Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317195740.xml (deflated 41%) 2022-03-17T22:12:57.9270325Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200306.xml (deflated 39%) 2022-03-17T22:12:57.9270815Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220317192805.xml (deflated 69%) 2022-03-17T22:12:57.9271324Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220317192805.xml (deflated 53%) 2022-03-17T22:12:57.9330719Z ##[group]Run actions/upload-artifact@v2 2022-03-17T22:12:57.9330933Z with: 2022-03-17T22:12:57.9331185Z name: test-reports 2022-03-17T22:12:57.9331434Z retention-days: 14 2022-03-17T22:12:57.9331692Z if-no-files-found: error 2022-03-17T22:12:57.9331968Z path: test-reports-*.zip 2022-03-17T22:12:57.9332153Z env: 2022-03-17T22:12:57.9332603Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:57.9333245Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:57.9333887Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:57.9334404Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:57.9334664Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:57.9334867Z IN_CI: 1 2022-03-17T22:12:57.9335079Z IS_GHA: 1 2022-03-17T22:12:57.9335285Z IN_WHEEL_TEST: 1 2022-03-17T22:12:57.9335743Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:57.9336170Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:57.9336401Z PR_LABELS: [] 2022-03-17T22:12:57.9337116Z GITHUB_TOKEN: *** 2022-03-17T22:12:57.9337401Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:57.9337618Z PR_NUMBER: 2022-03-17T22:12:57.9337929Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:57.9338186Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:57.9338451Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:57.9339213Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:57.9339686Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:57.9339942Z TEST_CONFIG: distributed 2022-03-17T22:12:57.9340169Z SHARD_NUMBER: 1 2022-03-17T22:12:57.9340383Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:57.9340597Z PR_BODY: 2022-03-17T22:12:57.9340952Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:57.9341523Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:57.9341742Z SHM_SIZE: 8g 2022-03-17T22:12:57.9341938Z ##[endgroup] 2022-03-17T22:12:58.0020021Z With the provided path, there will be 1 file uploaded 2022-03-17T22:12:58.0020896Z Starting artifact upload 2022-03-17T22:12:58.0022592Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-03-17T22:12:58.0022884Z Artifact name is valid! 2022-03-17T22:12:58.3622871Z Container for artifact "test-reports" successfully created. Starting upload of file(s) 2022-03-17T22:12:59.0830339Z Total size of all the files uploaded is 832671 bytes 2022-03-17T22:12:59.0831288Z File upload process has finished. Finalizing the artifact upload 2022-03-17T22:12:59.3920045Z Artifact has been finalized. All files have been successfully uploaded! 2022-03-17T22:12:59.3920693Z 2022-03-17T22:12:59.3921166Z The raw size of all the files that were specified for upload is 832671 bytes 2022-03-17T22:12:59.3922630Z The size of all the files that were uploaded is 832671 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-03-17T22:12:59.3923508Z 2022-03-17T22:12:59.3924987Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-03-17T22:12:59.3926050Z 2022-03-17T22:12:59.3926656Z Artifact test-reports has been successfully uploaded! 2022-03-17T22:12:59.4018770Z ##[group]Run set -x 2022-03-17T22:12:59.4019225Z set -x 2022-03-17T22:12:59.4019716Z python3 -m pip install -r requirements.txt 2022-03-17T22:12:59.4020305Z python3 -m pip install boto3==1.19.12 2022-03-17T22:12:59.4021012Z GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") 2022-03-17T22:12:59.4021667Z export GHA_WORKFLOW_JOB_ID 2022-03-17T22:12:59.4022311Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-03-17T22:12:59.4078796Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T22:12:59.4079276Z env: 2022-03-17T22:12:59.4079791Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:12:59.4080610Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:12:59.4081601Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:12:59.4082508Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:12:59.4083108Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:12:59.4083514Z IN_CI: 1 2022-03-17T22:12:59.4083875Z IS_GHA: 1 2022-03-17T22:12:59.4084248Z IN_WHEEL_TEST: 1 2022-03-17T22:12:59.4084759Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:12:59.4085399Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:12:59.4085940Z PR_LABELS: [] 2022-03-17T22:12:59.4086716Z GITHUB_TOKEN: *** 2022-03-17T22:12:59.4087149Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:12:59.4087580Z PR_NUMBER: 2022-03-17T22:12:59.4088010Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:12:59.4088484Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:12:59.4088931Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:12:59.4089734Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:12:59.4090647Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:12:59.4091195Z TEST_CONFIG: distributed 2022-03-17T22:12:59.4091613Z SHARD_NUMBER: 1 2022-03-17T22:12:59.4092000Z NUM_TEST_SHARDS: 1 2022-03-17T22:12:59.4092396Z PR_BODY: 2022-03-17T22:12:59.4092856Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:12:59.4093548Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:12:59.4094156Z SHM_SIZE: 8g 2022-03-17T22:12:59.4094524Z BRANCH: master 2022-03-17T22:12:59.4094889Z TAG: 2022-03-17T22:12:59.4095248Z WORKFLOW_ID: 2000264370 2022-03-17T22:12:59.4095648Z ##[endgroup] 2022-03-17T22:12:59.4168976Z + python3 -m pip install -r requirements.txt 2022-03-17T22:13:00.3239644Z Collecting astunparse (from -r requirements.txt (line 2)) 2022-03-17T22:13:00.5025488Z Using cached https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl 2022-03-17T22:13:00.5078671Z Collecting expecttest (from -r requirements.txt (line 3)) 2022-03-17T22:13:00.5679413Z Using cached https://files.pythonhosted.org/packages/f3/81/b5ce301b7eef591e9ed817365beaa958fe241465aacc2e97c88a28b7ff71/expecttest-0.1.3-py3-none-any.whl 2022-03-17T22:13:00.5698549Z Collecting future (from -r requirements.txt (line 4)) 2022-03-17T22:13:00.6212021Z Collecting numpy (from -r requirements.txt (line 5)) 2022-03-17T22:13:01.3361362Z Using cached https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl 2022-03-17T22:13:01.6457483Z Collecting psutil (from -r requirements.txt (line 6)) 2022-03-17T22:13:01.9371652Z Collecting pyyaml (from -r requirements.txt (line 7)) 2022-03-17T22:13:02.0727455Z Using cached https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl 2022-03-17T22:13:02.0876078Z Collecting requests (from -r requirements.txt (line 8)) 2022-03-17T22:13:02.1888609Z Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl 2022-03-17T22:13:02.2062846Z Collecting setuptools (from -r requirements.txt (line 9)) 2022-03-17T22:13:02.6460957Z Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl 2022-03-17T22:13:02.7009297Z Collecting six (from -r requirements.txt (line 10)) 2022-03-17T22:13:02.7709729Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-03-17T22:13:02.7729319Z Collecting types-dataclasses (from -r requirements.txt (line 11)) 2022-03-17T22:13:02.8482433Z Using cached https://files.pythonhosted.org/packages/5f/f2/6e7fa314e734020db09234ae12e8b673f659f549ae20fd1b3c93315201b8/types_dataclasses-0.6.4-py3-none-any.whl 2022-03-17T22:13:02.8498240Z Collecting typing_extensions (from -r requirements.txt (line 12)) 2022-03-17T22:13:02.8661912Z Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl 2022-03-17T22:13:02.8682626Z Collecting dataclasses (from -r requirements.txt (line 13)) 2022-03-17T22:13:02.8780605Z Using cached https://files.pythonhosted.org/packages/fe/ca/75fac5856ab5cfa51bbbcefa250182e50441074fdc3f803f6e76451fab43/dataclasses-0.8-py3-none-any.whl 2022-03-17T22:13:02.8800491Z Collecting wheel<1.0,>=0.23.0 (from astunparse->-r requirements.txt (line 2)) 2022-03-17T22:13:02.9753351Z Using cached https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl 2022-03-17T22:13:02.9810001Z Collecting urllib3<1.27,>=1.21.1 (from requests->-r requirements.txt (line 8)) 2022-03-17T22:13:03.0976713Z Using cached https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-py2.py3-none-any.whl 2022-03-17T22:13:03.1204182Z Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 8)) 2022-03-17T22:13:03.1941014Z Using cached https://files.pythonhosted.org/packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl 2022-03-17T22:13:03.1979411Z Collecting idna<4,>=2.5; python_version >= "3" (from requests->-r requirements.txt (line 8)) 2022-03-17T22:13:03.2615714Z Using cached https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-none-any.whl 2022-03-17T22:13:03.2656576Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->-r requirements.txt (line 8)) 2022-03-17T22:13:03.3411315Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-03-17T22:13:03.3465459Z Installing collected packages: wheel, six, astunparse, expecttest, future, numpy, psutil, pyyaml, urllib3, certifi, idna, charset-normalizer, requests, setuptools, types-dataclasses, typing-extensions, dataclasses 2022-03-17T22:13:05.4200982Z Successfully installed astunparse-1.6.3 certifi-2021.10.8 charset-normalizer-2.0.12 dataclasses-0.8 expecttest-0.1.3 future-0.18.2 idna-3.3 numpy-1.19.5 psutil-5.9.0 pyyaml-6.0 requests-2.27.1 setuptools-59.6.0 six-1.16.0 types-dataclasses-0.6.4 typing-extensions-4.1.1 urllib3-1.26.9 wheel-0.37.1 2022-03-17T22:13:05.5116608Z + python3 -m pip install boto3==1.19.12 2022-03-17T22:13:06.4193982Z Collecting boto3==1.19.12 2022-03-17T22:13:07.1110990Z Using cached https://files.pythonhosted.org/packages/5e/e1/156846b09fca21b9b164c54200011e3bd17f29187cbfc6903a8e0281a304/boto3-1.19.12-py3-none-any.whl 2022-03-17T22:13:07.1269313Z Collecting botocore<1.23.0,>=1.22.12 (from boto3==1.19.12) 2022-03-17T22:13:07.9927830Z Using cached https://files.pythonhosted.org/packages/6a/73/552b27e3a1b4f83630907c4958be78e9d4c906e73efd554ebd5e21cb1692/botocore-1.22.12-py3-none-any.whl 2022-03-17T22:13:08.3039747Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.19.12) 2022-03-17T22:13:08.3582969Z Using cached https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl 2022-03-17T22:13:08.3617839Z Collecting s3transfer<0.6.0,>=0.5.0 (from boto3==1.19.12) 2022-03-17T22:13:08.4223812Z Using cached https://files.pythonhosted.org/packages/7b/9c/f51775ebe7df5a7aa4e7c79ed671bde94e154bd968aca8d65bb24aba0c8c/s3transfer-0.5.2-py3-none-any.whl 2022-03-17T22:13:08.4292911Z Collecting urllib3<1.27,>=1.25.4 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-03-17T22:13:08.4791039Z Using cached https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-py2.py3-none-any.whl 2022-03-17T22:13:08.5018736Z Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-03-17T22:13:08.5494600Z Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl 2022-03-17T22:13:08.5571846Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-03-17T22:13:08.5757850Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-03-17T22:13:08.5780288Z Installing collected packages: urllib3, jmespath, six, python-dateutil, botocore, s3transfer, boto3 2022-03-17T22:13:09.1460581Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.9 2022-03-17T22:13:09.2402989Z ++ python3 .github/scripts/get_workflow_job_id.py 2000264370 worker-rocm-amd-34 2022-03-17T22:13:09.8497092Z + GHA_WORKFLOW_JOB_ID=5590691830 2022-03-17T22:13:09.8498918Z + export GHA_WORKFLOW_JOB_ID 2022-03-17T22:13:09.8500407Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-03-17T22:13:13.3522855Z [scribe] Scribe access token not provided, sending report via boto3... 2022-03-17T22:13:13.3526310Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:iam::308535385114:user/circleci_docker_builder is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:308535385114:function:gh-ci-scribe-proxy because no identity-based policy allows the lambda:InvokeFunction action 2022-03-17T22:13:13.3528657Z ERROR ENCOUNTERED WHEN UPLOADING TO S3: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied 2022-03-17T22:13:13.3529432Z 2022-03-17T22:13:13.3530010Z ----- Historic stats comparison result ------ 2022-03-17T22:13:13.3530485Z 2022-03-17T22:13:13.3531117Z job: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:13:13.3531956Z commit: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:13:13.3532402Z 2022-03-17T22:13:13.3532890Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-03-17T22:13:13.3533526Z 2022-03-17T22:13:13.3533736Z : (master) 2022-03-17T22:13:13.3534220Z | 2022-03-17T22:13:13.3534826Z * 4b4f652f79 (HEAD) total time 4011.38s 2022-03-17T22:13:13.3535464Z * 96ed3320f8 0 reports 2022-03-17T22:13:13.3536071Z * 119df90ead 0 reports 2022-03-17T22:13:13.3536673Z * d8c0f2f35d 0 reports 2022-03-17T22:13:13.3537288Z * 40a51edca7 0 reports 2022-03-17T22:13:13.3537932Z * f7fe75c766 0 reports 2022-03-17T22:13:13.3539018Z * 79c3bbceb2 0 reports 2022-03-17T22:13:13.3539654Z * 14dcb5a1a0 0 reports 2022-03-17T22:13:13.3540239Z * abc868dd64 0 reports 2022-03-17T22:13:13.3540840Z * fafa54a940 0 reports 2022-03-17T22:13:13.3541413Z * d67a265881 0 reports 2022-03-17T22:13:13.3541947Z | 2022-03-17T22:13:13.3542426Z : 2022-03-17T22:13:13.3542709Z 2022-03-17T22:13:13.3543090Z Removed (across 0 suites) 0 tests, totaling 0.00s 2022-03-17T22:13:13.3543932Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-03-17T22:13:13.3544757Z Added (across 48 suites) 977 tests, totaling +6296.38s 2022-03-17T22:13:13.4126259Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2022-03-17T22:13:13.4127334Z # ignore expansion of "docker ps -q" since it could be empty 2022-03-17T22:13:13.4128154Z # shellcheck disable=SC2046 2022-03-17T22:13:13.4129140Z docker stop $(docker ps -q) || true 2022-03-17T22:13:13.4129894Z # Prune all of the docker images 2022-03-17T22:13:13.4130599Z docker system prune -af 2022-03-17T22:13:13.4202920Z shell: /bin/bash -e {0} 2022-03-17T22:13:13.4203508Z env: 2022-03-17T22:13:13.4204319Z BUILD_ENVIRONMENT: linux-bionic-rocm4.5-py3.7-distributed 2022-03-17T22:13:13.4205604Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7 2022-03-17T22:13:13.4206824Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T22:13:13.4207912Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T22:13:13.4208907Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T22:13:13.4209545Z IN_CI: 1 2022-03-17T22:13:13.4210107Z IS_GHA: 1 2022-03-17T22:13:13.4210667Z IN_WHEEL_TEST: 1 2022-03-17T22:13:13.4211472Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T22:13:13.4212501Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T22:13:13.4213335Z PR_LABELS: [] 2022-03-17T22:13:13.4214472Z GITHUB_TOKEN: *** 2022-03-17T22:13:13.4215180Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T22:13:13.4215887Z PR_NUMBER: 2022-03-17T22:13:13.4216559Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T22:13:13.4217302Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T22:13:13.4218017Z GIT_DEFAULT_BRANCH: master 2022-03-17T22:13:13.4218804Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:13:13.4219359Z JOB_BASE_NAME: linux-bionic-rocm4.5-py3.7-distributed-test 2022-03-17T22:13:13.4219681Z TEST_CONFIG: distributed 2022-03-17T22:13:13.4219906Z SHARD_NUMBER: 1 2022-03-17T22:13:13.4220134Z NUM_TEST_SHARDS: 1 2022-03-17T22:13:13.4220344Z PR_BODY: 2022-03-17T22:13:13.4220609Z DOCKER_HOST: unix:///run/user/1123/docker.sock 2022-03-17T22:13:13.4220988Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2022-03-17T22:13:13.4221326Z SHM_SIZE: 8g 2022-03-17T22:13:13.4221542Z ##[endgroup] 2022-03-17T22:13:13.9020880Z 38dccec0eac5 2022-03-17T22:13:22.4001769Z Deleted Containers: 2022-03-17T22:13:22.4002350Z 38dccec0eac565a8c82df885e30ce5584a016dd65e3050f93bf07338c6b019b2 2022-03-17T22:13:22.4002600Z 2022-03-17T22:13:25.8593457Z Deleted Images: 2022-03-17T22:13:25.8595765Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T22:13:25.8598344Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-rocm4.5-py3.7@sha256:a482b3a3008d87393da82f727e1eccc5f8ab041e50908ba7de84e9561c04f097 2022-03-17T22:13:25.8599849Z deleted: sha256:c4c0740221d96a0e7524adc8d79dc9c73576ae2af130ccd7c60036ae36cba414 2022-03-17T22:13:25.8600928Z deleted: sha256:da1d40f8b79ba70f87cf47b6bef242dd5f83d8d2d1093b88851c0d96954804c3 2022-03-17T22:13:25.8602138Z deleted: sha256:459e3206b4c9916849f738d66ea6fa892356e28e698fa431856850f693066a18 2022-03-17T22:13:25.8603125Z deleted: sha256:a6086582d802e6d00e1e0c1b3a105af27dc6723f616a02b083d8e3e23e7297c9 2022-03-17T22:13:25.8604163Z deleted: sha256:a2bf747064fcbd2d4f1bd503d40e5b6866d62f5082662b0b077b511d4cb1b31b 2022-03-17T22:13:25.8605244Z deleted: sha256:58d935d3ece5bea37ffc91008d40cb3a3cd7da2789164d88786241163142ab07 2022-03-17T22:13:25.8606360Z deleted: sha256:6c9d70a1bcb75a9fafba2c94101dec9a746c2b3bbd38ae889ea0afe5b94f777b 2022-03-17T22:13:25.8607437Z deleted: sha256:008929403f5d3cc9d8d266a0a6312bc4adf4a963602714a5c52a251c365b9802 2022-03-17T22:13:25.8609006Z deleted: sha256:07c99bd59ef607b3e22e7a1b9017efd6d6c45d4311f722a11b23548139c259c6 2022-03-17T22:13:25.8610079Z deleted: sha256:d351287ff2653cabd3a9df086e49182c5b2be94b678d3b34917c0925aad0f5e2 2022-03-17T22:13:25.8611146Z deleted: sha256:ab7ca3af35a71e66b74097a340d66a5fa05b4d0f05814f5dd16db83aaa3f9f79 2022-03-17T22:13:25.8612412Z deleted: sha256:22186d3c43692a24cf3b6bd0eae6196bb0c38edb76c4b37a29faf0104d438e1a 2022-03-17T22:13:25.8613487Z deleted: sha256:72d9b489e5e56d1b8fc20abebd43ec20a515562e6102600d8d31fa946636206c 2022-03-17T22:13:25.8614551Z deleted: sha256:cd5bdc7c5dbe53853cc6303c31a78d5c55956807e245e7edda3a62f362f4c5f1 2022-03-17T22:13:25.8615640Z deleted: sha256:92acb454250b968d843aadce94240c991b2e5dcec664912bae07f0f1649ccbba 2022-03-17T22:13:25.8616666Z deleted: sha256:c1196c37ba93b1908548a72170313514fbc88dea32aac12da5cc82c9a6f2f3cf 2022-03-17T22:13:25.8617679Z deleted: sha256:6bd1dbc812a044af5222a368d3c8758a11a4e0ef5193d4bc012d47fcbfae0dec 2022-03-17T22:13:25.8618715Z deleted: sha256:c042fed7e6f09a16886f8cc683ef51e094f4d3cc3874cc35092545ad880da88a 2022-03-17T22:13:25.8619786Z deleted: sha256:80ed8e2f652a66ee4642cde4f98bda2d737efbf6e8018fe94030511681bb9aad 2022-03-17T22:13:25.8620869Z deleted: sha256:bcbc27ea46aa938f275a98579233b25beffa7e6ce2b4bb246363b9fa327fa290 2022-03-17T22:13:25.8621967Z deleted: sha256:5ce22666d3f6f5c996964d2161a2e74cd17a9b27b003013c4edd15e5b56aaa31 2022-03-17T22:13:25.8623001Z deleted: sha256:77cf1b8cd6e7907b901fe6a5f7d2f7cf49a07c87496dc8c23c456d63412154cf 2022-03-17T22:13:25.8624034Z deleted: sha256:144be487e806e26c4d8e84d7b1b580cc157c36fb832cef8084f463758e13757f 2022-03-17T22:13:25.8624971Z deleted: sha256:66a648d293d079d598e204797082765180289fc4afdd553071173418c3b3025b 2022-03-17T22:13:25.8625959Z deleted: sha256:82efdb23840db8c15c6b63f963626a9c1ce9fa51e5ae7823723535cbbca7a401 2022-03-17T22:13:25.8627027Z deleted: sha256:f5c0bee9dda1a9fde01ec93b4b1b86145553a45066a41466b0cb0bbc4338b7c5 2022-03-17T22:13:25.8628118Z deleted: sha256:9c6c4216c2b845fc0bd45a9260e7a1fcdd430e8ccc4adb30cad566bace5978ab 2022-03-17T22:13:25.8629164Z deleted: sha256:58d72360b476b9e6c050a1818a862d6d7f77e53502be15668ebc9eda225475ad 2022-03-17T22:13:25.8630184Z deleted: sha256:b8c9b645e3dd51d1db660b9f01abd4ca8697a007561ac349862850c4bb913cd1 2022-03-17T22:13:25.8631227Z deleted: sha256:a4e25480be6bd70ea8f073f5af661a58e498b79f81d4d17a004bcbe54c7bd14d 2022-03-17T22:13:25.8631824Z 2022-03-17T22:13:25.8632113Z Total reclaimed space: 24.53GB 2022-03-17T22:13:25.8716079Z Post job cleanup. 2022-03-17T22:13:25.9755371Z [command]/usr/bin/git version 2022-03-17T22:13:25.9803907Z git version 2.35.1 2022-03-17T22:13:25.9845560Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-03-17T22:13:25.9895144Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-03-17T22:13:26.0261126Z Entering 'android/libs/fbjni' 2022-03-17T22:13:26.0328109Z Entering 'third_party/FP16' 2022-03-17T22:13:26.0395291Z Entering 'third_party/FXdiv' 2022-03-17T22:13:26.0463230Z Entering 'third_party/NNPACK' 2022-03-17T22:13:26.0533071Z Entering 'third_party/QNNPACK' 2022-03-17T22:13:26.0592053Z Entering 'third_party/XNNPACK' 2022-03-17T22:13:26.0682944Z Entering 'third_party/benchmark' 2022-03-17T22:13:26.0745166Z Entering 'third_party/breakpad' 2022-03-17T22:13:26.0798763Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T22:13:26.0868979Z Entering 'third_party/cpuinfo' 2022-03-17T22:13:26.0935939Z Entering 'third_party/cub' 2022-03-17T22:13:26.0999061Z Entering 'third_party/cudnn_frontend' 2022-03-17T22:13:26.1073139Z Entering 'third_party/eigen' 2022-03-17T22:13:26.1144039Z Entering 'third_party/fbgemm' 2022-03-17T22:13:26.1204219Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T22:13:26.1270873Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T22:13:26.1342560Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T22:13:26.1415208Z Entering 'third_party/flatbuffers' 2022-03-17T22:13:26.1481332Z Entering 'third_party/fmt' 2022-03-17T22:13:26.1552493Z Entering 'third_party/foxi' 2022-03-17T22:13:26.1608491Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T22:13:26.1671838Z Entering 'third_party/gloo' 2022-03-17T22:13:26.1744599Z Entering 'third_party/googletest' 2022-03-17T22:13:26.1806459Z Entering 'third_party/ideep' 2022-03-17T22:13:26.1868462Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T22:13:26.1938963Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T22:13:26.2005262Z Entering 'third_party/ios-cmake' 2022-03-17T22:13:26.2062832Z Entering 'third_party/kineto' 2022-03-17T22:13:26.2131816Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T22:13:26.2196738Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T22:13:26.2261445Z Entering 'third_party/nccl/nccl' 2022-03-17T22:13:26.2323086Z Entering 'third_party/neon2sse' 2022-03-17T22:13:26.2393861Z Entering 'third_party/onnx' 2022-03-17T22:13:26.2478584Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T22:13:26.2548840Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T22:13:26.2621625Z Entering 'third_party/onnx-tensorrt' 2022-03-17T22:13:26.2684715Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T22:13:26.2761704Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T22:13:26.2829051Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T22:13:26.2897445Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T22:13:26.2965909Z Entering 'third_party/pocketfft' 2022-03-17T22:13:26.3029863Z Entering 'third_party/protobuf' 2022-03-17T22:13:26.3102805Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T22:13:26.3166894Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T22:13:26.3238652Z Entering 'third_party/psimd' 2022-03-17T22:13:26.3303237Z Entering 'third_party/pthreadpool' 2022-03-17T22:13:26.3358923Z Entering 'third_party/pybind11' 2022-03-17T22:13:26.3425225Z Entering 'third_party/python-enum' 2022-03-17T22:13:26.3487252Z Entering 'third_party/python-peachpy' 2022-03-17T22:13:26.3551239Z Entering 'third_party/python-six' 2022-03-17T22:13:26.3620762Z Entering 'third_party/sleef' 2022-03-17T22:13:26.3673588Z Entering 'third_party/tbb' 2022-03-17T22:13:26.3747801Z Entering 'third_party/tensorpipe' 2022-03-17T22:13:26.3812443Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T22:13:26.3872476Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T22:13:26.3933333Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T22:13:26.4000329Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T22:13:26.4061647Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T22:13:26.4135906Z Entering 'third_party/zstd' 2022-03-17T22:13:26.4221035Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-03-17T22:13:26.4288277Z http.https://github.com/.extraheader 2022-03-17T22:13:26.4306906Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-03-17T22:13:26.4360604Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-03-17T22:13:26.4731388Z Entering 'android/libs/fbjni' 2022-03-17T22:13:26.4766313Z http.https://github.com/.extraheader 2022-03-17T22:13:26.4819224Z Entering 'third_party/FP16' 2022-03-17T22:13:26.4854049Z http.https://github.com/.extraheader 2022-03-17T22:13:26.4913904Z Entering 'third_party/FXdiv' 2022-03-17T22:13:26.4947050Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5006370Z Entering 'third_party/NNPACK' 2022-03-17T22:13:26.5036817Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5097261Z Entering 'third_party/QNNPACK' 2022-03-17T22:13:26.5125881Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5177009Z Entering 'third_party/XNNPACK' 2022-03-17T22:13:26.5210944Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5290295Z Entering 'third_party/benchmark' 2022-03-17T22:13:26.5322556Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5383694Z Entering 'third_party/breakpad' 2022-03-17T22:13:26.5416512Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5477181Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T22:13:26.5509306Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5564995Z Entering 'third_party/cpuinfo' 2022-03-17T22:13:26.5596940Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5656018Z Entering 'third_party/cub' 2022-03-17T22:13:26.5686352Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5745034Z Entering 'third_party/cudnn_frontend' 2022-03-17T22:13:26.5781239Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5849327Z Entering 'third_party/eigen' 2022-03-17T22:13:26.5884788Z http.https://github.com/.extraheader 2022-03-17T22:13:26.5948783Z Entering 'third_party/fbgemm' 2022-03-17T22:13:26.5985719Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6040138Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T22:13:26.6067492Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6117994Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T22:13:26.6153429Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6205711Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T22:13:26.6234002Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6300338Z Entering 'third_party/flatbuffers' 2022-03-17T22:13:26.6337737Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6395189Z Entering 'third_party/fmt' 2022-03-17T22:13:26.6432737Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6487319Z Entering 'third_party/foxi' 2022-03-17T22:13:26.6518596Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6575730Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T22:13:26.6606846Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6665324Z Entering 'third_party/gloo' 2022-03-17T22:13:26.6702936Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6762316Z Entering 'third_party/googletest' 2022-03-17T22:13:26.6800091Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6845578Z Entering 'third_party/ideep' 2022-03-17T22:13:26.6881074Z http.https://github.com/.extraheader 2022-03-17T22:13:26.6937285Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T22:13:26.6973191Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7032677Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T22:13:26.7064975Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7132482Z Entering 'third_party/ios-cmake' 2022-03-17T22:13:26.7169237Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7232106Z Entering 'third_party/kineto' 2022-03-17T22:13:26.7270327Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7327254Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T22:13:26.7364524Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7417842Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T22:13:26.7451440Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7517855Z Entering 'third_party/nccl/nccl' 2022-03-17T22:13:26.7555371Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7607530Z Entering 'third_party/neon2sse' 2022-03-17T22:13:26.7643405Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7698858Z Entering 'third_party/onnx' 2022-03-17T22:13:26.7729246Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7816544Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T22:13:26.7844280Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7901555Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T22:13:26.7937865Z http.https://github.com/.extraheader 2022-03-17T22:13:26.7993675Z Entering 'third_party/onnx-tensorrt' 2022-03-17T22:13:26.8030937Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8090446Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T22:13:26.8126066Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8198682Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T22:13:26.8235198Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8287768Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T22:13:26.8325068Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8386292Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T22:13:26.8418994Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8491028Z Entering 'third_party/pocketfft' 2022-03-17T22:13:26.8526866Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8579826Z Entering 'third_party/protobuf' 2022-03-17T22:13:26.8618363Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8685429Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T22:13:26.8722798Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8784244Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T22:13:26.8818104Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8882483Z Entering 'third_party/psimd' 2022-03-17T22:13:26.8916169Z http.https://github.com/.extraheader 2022-03-17T22:13:26.8968855Z Entering 'third_party/pthreadpool' 2022-03-17T22:13:26.9003355Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9062580Z Entering 'third_party/pybind11' 2022-03-17T22:13:26.9093007Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9154204Z Entering 'third_party/python-enum' 2022-03-17T22:13:26.9190170Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9242222Z Entering 'third_party/python-peachpy' 2022-03-17T22:13:26.9277735Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9336240Z Entering 'third_party/python-six' 2022-03-17T22:13:26.9369849Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9426082Z Entering 'third_party/sleef' 2022-03-17T22:13:26.9454877Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9506942Z Entering 'third_party/tbb' 2022-03-17T22:13:26.9539481Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9600045Z Entering 'third_party/tensorpipe' 2022-03-17T22:13:26.9630832Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9681066Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T22:13:26.9717027Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9777755Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T22:13:26.9811488Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9858434Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T22:13:26.9894903Z http.https://github.com/.extraheader 2022-03-17T22:13:26.9953365Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T22:13:26.9989852Z http.https://github.com/.extraheader 2022-03-17T22:13:27.0040654Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T22:13:27.0076094Z http.https://github.com/.extraheader 2022-03-17T22:13:27.0143364Z Entering 'third_party/zstd' 2022-03-17T22:13:27.0173290Z http.https://github.com/.extraheader 2022-03-17T22:13:27.0543718Z Cleaning up orphan processes